Contextually intelligent communications systems and processes

ABSTRACT

A contextually intelligent communication system and processes acquire and processes data on the current context of a user who is using a connected mobile communication device such as a smart phone or tablet by using various sensors, image recognition or augmented reality residing in the connected device; providing additional data to define the user&#39;s current environment; combining the contextual data on the device with the additional data to define the user&#39;s complete, relevant context; gathering the user&#39;s current context data and updating the device by uploading the data via a wide area mobile network to a contextually intelligent server in the cloud, matching the user&#39;s current contextual data with the user&#39;s past and historical data and then downloading updated data back to the device, with the updated data including any of a variety of contextually relevant information such as feedback, experiences, recommendations, offers, coupons, advice, tactile feedback, content such as visual and audio representations, augmented reality, and other audio/visual displays to the device of the user that is predictably useful and relevant to the user&#39;s current context and future context as the user enters a new context.

CROSS REFERENCE TO RELATED APPLICATION

The present application claims the benefit of priority of U.S.provisional patent application 61/709,710, filed Oct. 4, 2012, and ofU.S. provisional patent application 61/882,593, filed Sep. 25, 2013.

FIELD OF INVENTION

The presently described systems and processes relate to the field ofcontextually intelligent communications that acquire and process datarelated to a user who is using or connected to a mobile communicationdevice.

BACKGROUND

High tech mobile chip companies like Qualcomm and Broadcom, mobiledevice manufacturers such as Apple, HTC, Nokia and Samsung, mobilenetwork operators such as AT&T, Verizon and Orange, mobile appdevelopers and publishers such as Trigger and thousands of others build,develop, create, invent, manufacture and/or publish products andprocesses in the field of electronic or telecommunications. Augmentedreality (AR) software solutions such as Qualcomm's Vuforia™ brandsoftware application and others such as Aurasma, image recognition (IR)solutions such as Qualcomm's Gimbal™ application, peer-to-peercommunication solutions such as Google's NFC application and Qualcomm' sAllJoyn™ brand application, mobile payment solutions such as Qualcomm'sPay™ brand application and Google's Wallet™ brand application, andcontext awareness solutions such as Qualcomm's Gimbal™ brand applicationalso play a role in the field of electronic communications and mobileapplications. Sensor fusion companies such as Virtual Eye, stadium andvenue solutions such as provided and developed by Cisco also play a rolein the field of electronic communications and mobile applications.Customer relationship management (CRM) systems hosted in the cloud areoffered by companies such as Salesforce.com, Oracle and Parallel6.Research facilities such as PARC, MIT and Qualcomm continue to fund andconduct research and development in many of the facets and features ofthe field of electronic communications.

The present architecture, systems and processes do not depend on anyspecific chip, mobile device, technology, operator or user. They workregardless of what AR or IR technology, CRM or sensor fusion system isused.

The present architecture, systems and processes are based on combiningcurrently existing technologies and blending them with traditionalmarketing and sales methodologies to form integrated solutions. Theywork with known technologies, and are adapted for and are adaptableexpected new technologies, devices and use-cases as they becomecommercially available.

This is the first time a simple to manage solution has been created bycombining complex and cutting edge technologies, combining it withreal-time contextual data, including past historical data on user'sbehavior to present solutions, offers and assistance to the end userbased on their current context and situation, past behavior andpreferences and predictable next actions and desires.

This overview will present a summary of the state of the art of thevarious technologies described above.

The present description details numerous embodiments and exemplary,detailed use cases. There are numerous combinations and sub-combinationfor the presently described systems and processes.

Numerous patents exist in each of the sub categories described below.The presently describe systems and processes is not dependent upon anyspecific such patent. Typically providers of technologies such asQualcomm's Vuforia SDK allows for patents to be filed based on theirunderlying technology.

Context Awareness

Contact awareness is a well-known discipline, and has significant,current R&D in progress. There is a lot of R&D in this space, atuniversities (MIT), research facilities (PARC, SRI, Freuenheuser) andcommercial corporations (Qualcomm, Apple). Qualcomm has just released abeta version of a SDK called “Gimbal”, which is the closest to theinvention. It features aspects of the invention, but does not includesensor fusion with other devices, input from other devices and does notinclude a commercial CRM back end that enables integration withhistorical demographic data as a stand-alone solution.

Context Intelligence

There is a lot of research in this space as well, but no knowncommercial solution available at this time. Same entities as above arethe strong players in this space.

Mobile Marketing

Numerous companies are doing marketing to the mobile devices. Some uselocation based services, others include social media status andintegrations, and others use a CRM back-end. No one have a completesolution that enables a brand to gather real time context information,combine it with historical data and automatically deliver messages tothe mobile device based on these.

AR

Numerous companies have technologies in this space, ranging from Layarwhich started using GPS for AR, to Nokia's new solution for Windows 7phones, to Sony, Blippar, Aurasma, Dekko and numerous others that havetheir own technology solution. The biggest player in the space isQualcomm with its Vuforia SDK. No one use AR to discover and verifycontext, and no one has yet to include a marketing back-end CRM systemand enable brands to have their own solution. The invention can use anyof the AR technologies mentioned.

IR

Image recognition is an aspect of AR, and many of the AR technologiesuse their own algorithms to create the AR experiences. Qualcomm' sGimbal SDK offers a free IR component. It does not include sensor inputfrom devices in proximity, or a marketing CRM back-end database.

Location Based Services

Many companies are offering Location Based Services (Placecast, Place-IQand many others) that feature aspects of the invention. None hasintegrated Context and sensor input.

Coupon Delivery

Groupon and numerous others have various solutions for delivery ofcoupons and offers. None have a complete solution like the inventionyet.

CRM

Salesforce.com, Oracle, Parallel6 and numerous others have cloud basedCRM systems. The invention is the first to include complex re-timecontextual information and combining it with historical anddemographical data.

Push Technology

There are numerous solutions that enable messages to be pushed to amobile device—some are fairly sophisticated.

Mobile Payment Solutions

Google Wallet, Qualcomm's Pay and numerous others are working on varioussolutions for mobile payments. None have full integrated with a cloudbased CRM solution like the invention.

Mobile Devices

The invention is not dependent upon any specific mobile device, andanticipates that the form factors of mobile devices will changeradically with new system and devices developed.

Peer-to-peer Communication

Numerous companies are working on standards and new means forpeer-to-peer communications, with Qualcomm's open source and free SDK“AllJoyn” being the most diversified cross platform, cross standards andcross device solution.

Stadium/Venue Solutions

Cisco, Virtual Eye and Sporting Innovation have a series of solutionsfor venue enhancements. None include the total solution for the venueand sport team owners as the invention does.

Sensor Fusion

Numerous companies spend a lot of money and resources on R&D for sensorfusion, with many starting to present and sell SDK solutions. Theseinclude Kinect, Qualcomm, Nokia and many others.

Ball and Player Tracking

Virtual Eye and Hawkeye have solutions that can track and verify balltrajectories in sports such as soccer, baseball, tennis and golf. Noneare using this information to drive AR experiences as the inventioncovers.

There is therefore a need to provide a method that allows establishing auser's context, create a definition of context and contextual awareness,communicate this to a database of known past behavior, combine these andthen deliver content from the cloud directly to the user's connecteddevice(s) to enable delivery of content that is relevant, appropriateand helpful for the end user, and present this at the right time andlocation, thus creating contextual wisdom creating a contextuallyintelligent communication solution

SUMMARY

The presently described architecture, systems and processes are directedto an electronic eco-system that is built upon a combination of diversetechnologies, and together is sometimes referred to as a “solution”.While substantial R&D efforts are underway globally in the fields ofelectronic, wireless and mobile communications the present system isbelieved to be the first such eco-system that is built upon acombination of diverse technologies. Qualcomm's technologies are offeredas licensed and open source code, with the ability to create IP that issupported by their technologies without restrictions as long as theirunderlying technology is not changed in any way.

The present systems and processes provide a complete end-to-endcommunication platform that includes a series of currently, butseparately existing technologies. The inventive combination of thefeatures wherein mobile communication devices becomes sensor gatheringdevices that determine the user's current context, then report thiscontext to a customer relationship management system and communicationplatform residing in the cloud, combine it with historical demographicdata then enables communication back to the users mobile devices,presenting the right message and content to the right user at the righttime and location while the user is in a specific context does currentlynot exist.

The presently described embodiments are simple to manage and combinecomplex and cutting edge technologies with real-time contextual dataand, optionally with past historical data on a user's behavior. Thecombined data is used to present solutions, offers and assistance to theend user based on the user's current context and situation, pastbehavior and preferences and predictable next actions and desires.

The presently described architecture, systems and processes comprise apreferred contextually intelligent communication platform (CICP) systemand a variety of preferred methods for its use. In one aspect the CICPis used for acquiring information from various sources to facilitate arecognition of one or more objects or images using sensor datatransmitted to an augmented reality (AR) system mobile applicationsoftware and/or an image recognition (IR) mobile application software.Using a camera of a mobile device with the AR and or IR application andthe device's camera, the recognized image or object is matched with anexisting data file for a known image or object preferably residing in animage/object server in the cloud. A server in the cloud is hereindefined as a computer server located elsewhere, hosted and managed bysome entity, such as for example Amazon's web services. The AR system ofa CICP can be used as a method for delivering an augmentation to a userof the CICP. An Augmentation is the overlay of a digital experience suchas an animated or still 3-D model, an animation sequence, a videooverlay, link to another website or information superimposed on the ARtarget as seen on the mobile device display screen. Representative,exemplary sources for an AR system include Vuforia™ brand softwareavailable from Qualcomm, Inc., San Diego, Calif. IR applications arealso, preferable used to recognize an object, match the image with aknown image in the cloud based image server and present an overlay ofinformation superimposed on the image and displayed on the mobile devicedisplay screen. Several preferred commercially available IR applications“Gimbal”, available from Qualcomm Labs, Inc., San Diego, Calif. Aconnected device is preferably a mobile communication device such as asmart phone or a tablet, and also includes other electroniccommunication devices such as desktop computers, laptops, AR glasses orcontact lenses, and the like. Such hardware may be connected to acommunications system such as for example, the internet, satellitecommunications, mobile telephone systems, Wi-Fi and the like. Thepresent system is adapted to process various information obtained fromthe sensor activity(s) including for example, a location, a time, anactivity, a speed and direction of travel, an anxiety level assessment,a social media status, a gender, a relationship status, an ambient soundor a light level, an air quality, or various other environmental aspectsof the user. The gathered and shared information flows through thesystem and is used intelligence activity by which the CICP processes aplurality of data files relating to a contextual environment of the userof a connected device. The information acquired by the CICP about aparticular user has immediate and future benefits for that user. Thecontextual environment of a user of a connected device is defined as theenvironmental circumstances that form a setting for an event, astatement or an idea for the user and in terms in which the user'senvironment can be understood and assessed by the user. This includesdiscovery of other devices and people in proximity, detection of anysensor system embedded in the physical world around the user, any sensorsystem worn or embedded in other persons or other devices or equipment.It also includes the time, day and location as well as the activity inwhich the user is currently engaged. The contextual environment of theuser of a connected device is termed and defined as a context bubble.The context bubble of a user of a connected device influences the useras a bubble of influence. As other people or devices enter thiscontextual bubble of influence, either by moving into it, or the usermoving that the user's bubble of influence encompasses the new entity(s)within the user's bubble, these devices will be detected, acknowledgedand contact established when appropriate.

In a second aspect, the CICP is used as a method for predicting a futurecontextual environmental setting for the user of the connected device.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing aspects and the attendant advantages of the presentsystems and processes will become more readily appreciated by referenceto the following detailed description, when taken in conjunction withthe accompanying drawings, wherein:

FIG. 1 is a schematic overview of a preferred embodiment eco-systemarchitecture including a context or virtual bubble;

FIG. 2 is a schematic representation of the FIG. 1 showing preferredexemplary connected devices;

FIG. 3 is a listing of exemplary, preferred categories of variables andinformation that may be collected and used to determine current contextfor the FIG. 1 context bubble;

FIG. 4 is a listing of exemplary, preferred categories of variables andtypes of information that may be collected and used to define thecurrent context and create current content awareness;

FIG. 5 is a listing of exemplary, preferred categories of variables andtypes of information that may be collected and used to determine pastbehavior and action;

FIG. 6 is a listing of exemplary, preferred variables and questions thatmay be posted, collected and used to determine the predictions and whatto present next;

FIG. 7 is a listing of exemplary content, experiences and informationthat may be gathered, stored and used on the content and experiencesdatabase of the FIG. 1 embodiment;

FIG. 8 is a schematic presentation of delivery of content from a cloudbased customer relationship management system to the FIG. 1 contextbubble with a feedback loop to determine the usefulness of the contentdelivered;

FIG. 9 is a schematic presentation of two users who are connected witheach other at the intersection of their respective context bubbles andhow devices that are in their context bubbles, respectively canestablish peer-to-peer communication;

FIG. 10 is a schematic presentation of a mobile communication devicewith installed software that enables image recognition through itscamera and how this can establish and verify that the user of the mobiledevice is interacting with a real life object, image or video/audio feedthat is pre-captured and stored in a cloud based target and imagerecognition database, while connected to a cloud based or in adevice-augmented reality experience, and that the whole experience andactivity is recorded, reported and captured on a cloud based customerrelationship management system that may be integrated with the FIG. 1embodiment;

FIG. 11 is a schematic presentation of an alternate embodimentcommunication system and information flow process that captures contextinformation on a connected device, combines an augmented reality overlaydisplayed on the device and superimposed on a product, aggregates andreports this information from the mobile device to a cloud baseddatabase system that combines analytics, storage, processing andfiltering, combining demographic past behavior data, combines this andmatch it against relevant content, advertisements and offers into amobile customer relationship management system that mines, analyses,search and enrich the data, then based on the collective informationpresents relevant content back to the connected device;

FIG. 12 is a schematic presentation of preferred system that includesexemplary servers that report to the mobile customer relationshipmanagement system including the context input detected, recorded andreported from a connected device and then how the augmented relationshiptarget database server, content database server, demographic andhistorical database server and the past and inferred historical andpreference database server provides input to the mobile customerrelationship management system, and the content server is presentedinformation and content from advertisement database servers, coupondatabase servers and information database servers, and how a preferredmobile customer relationship management system combines all of the aboveto present the relevant information, content and offers directly to aspecific mobile device;

FIG. 13 is a schematic presentation on how the FIG. 12 system interactswith other devices, people, other people's devices, equipment, point ofsale (POS) systems, ticket and order systems to deliver content, offers,services and information that is relevant to the mobile device and theitems listed that is in its proximity;

FIG. 14 is a schematic presentation on how the system described in FIG.12 can become an intelligent and self-learning and continually improvingsystem by providing feedback from the connected device recording theeffectiveness of the offers, coupons and information presented to it bythe system in FIG. 12, report this into a feedback loop and store thisinto a result database server which reports this into a database thatrecords this information and combines it with past recorded data andthen reports this back into the historic and preference database in thesystem depicted in FIG. 12 providing a complete feedback loop with eachinteraction recorded and stored for repeated use when the system isactivated again thereby creating self-learning;

FIG. 15 is a schematic overview of the information flow from contextdevices such as sensors and infrastructures, past and historical data,combined with context that is detected and discovered with historicaldata, inference and predicted desired results and personalized offersare flowing in the path of information gathering first, then matchingand refining the data, then present it to the device thus creatingcontextual wisdom;

FIG. 16 is a schematic presentation on how the system described in FIG.14 can become an intelligent, self-learning and continually improvingsystem providing contextually intelligent communication by includingcollection information such as contextual data and input from devices inthe proximity of the mobile device as shown in FIG. 13, gather this byblending this with context input as shown in FIG. 12, match it with pastpreferences and historical data as described in FIG. 14, adapting andinferring contextual wisdom by refining it further as described in FIG.14, following the path outlined in FIG. 15, then present the relevantinformation to the device at the right time, location, while the deviceis in the right and known proximity to other devices, the right contextand the content served is relevant to the user of the device based onthe users demographic information, past behavior, learned and capturedbehavior and preferences, thus creating a contextually intelligentcommunication;

FIG. 17 depicts the steps used to gather data, combine it withinformation, blend past knowledge, combine to achieve an understandingof the user to derive at wisdom about the user to enable the delivery ofthe right content to the right user at the right time and place;

FIG. 18 provides an overview of representative entities and industriesthat may benefit from use of the invention, where some will provideinformation and technologies, while others will be using the system fordelivery of messages to their patrons, customers, visitors and clients;

FIG. 19 provides a schematic overview of how the present system can beused by a user to detect and verify the broadcast TV, film, movie orinternet content the user is watching, and how the system can thenpresent relevant, appropriate and supplementary content to the user'sconnected mobile device based on the content the user is consuming orinteracting with;

FIG. 20 provides an overview of the user with a defined context bubbleas the user enters a location that has established a defined geo-fenceand how the event of crossing the border is recorded by the customerrelationship management system, which then connects with the contentdatabase to present the appropriate content to the user based on thecontent associated with the location that has established the geo-fence;

FIG. 21 provides a schematic overview of how the FIG. 10 system isadapted for use to detect, record and report real-life content, imagesand objects the user of the mobile device and how this can be enhancedby capturing data from a sports event and using this combinedinformation as input to control an augmented reality experience that issuperimposed on the content, image or real life object;

FIG. 22 provides a schematic overview of how the FIG. 20 system isfurther adapted for capturing information from sensors in a sportingevent, sensors on the field and sensors on players, their equipment andprotective gear, then gathering this information and providing it asinput for playback on devices that can provide tactile feedback to theuser of the mobile device and the connected playback devices;

FIG. 23 provides an overview of a system embodiment adapted for use by astadium owner or operator where the system determines the user's intentto attend an event at the stadium, verifies and records thisinformation, then enables communication of relevant offers to the useras the user approaches the stadium on the way to the event, how the usercan be engaged, encouraged to participate, entertained and rewarded forsharing the experience with others while at the stadium, thenacknowledges and thanks the user for attending as the user is leaving,then rewarded, enticed and promoted to return for another future eventwhen returning home;

FIG. 24 provides a version of the FIG. 23 system that is customized fora theme park owner where in addition to the features in the FIG. 23system the system is adapted to entertain visitors to the park whilethey are waiting in lines to go on rides, and for the parkowner/operator to entice the visitors to remain in the park as long aspossible by providing time enticements, such as restricted treasurehunts and scavenger hunt games;

FIG. 25 shows the technologies and sensor input needed at the variousstages of the FIGS. 23 and 24 embodiments;

FIG. 26 is a schematic overview of an exemplary flow of actions andevents that occur when a mobile device has a treasure hunt applicationwith augmented reality and geographically located triggers, targets andpuzzles;

FIG. 27 is a schematic overview of an exemplary information flow for ahealth benefit embodiment is which a user wearing a sensor devicecommunicates with the user's connected mobile device and the sensordevice measures health and medical data, such as blood pressure, bloodsugar levels, cholesterol and heart rate for example.

FIG. 28 provides a schematic depiction of the system adapted for use ina medical application wherein the device captures, records and transmitsmedical and health data, as further shown and described with respect tothe FIG. 27 embodiment.

FIG. 29 is a schematic depiction of the system adapted for use tocollect and process health data before and after the consumption of ahealth product.

FIG. 30 is a schematic, exemplary depiction of how data collected isstored on the contextual database, may be combined and accessed byselect individuals and organizations, and further used.

FIG. 31 is a schematic depiction overview of processing steps andinformation flow paths of the FIGS. 27, 28, 29 and 30 embodiments.

FIG. 32 is a schematic depiction of a preferred mobile customerrelationship management system (MCRM) embodiment that is customized foran organization that operates a franchise business with numerousfranchisees at various physical locations.

FIG. 33 is a schematic depiction of a preferred mobile customerrelationship management system (MCRM) customized for a sport league thatoperates a franchise-like business with numerous sport team franchiseesat various physical locations.

FIG. 34 is a schematic depiction of how the distance between a devicethat displays an AR or IR overlay and the physical target that triggersthe experience can control the volume of audio included in the ARexperience or the strength and intensity of any tactile feedbackincluded in the experience.

DETAILED DESCRIPTION

The present inventions relate to contextually intelligent communicationarchitecture, platforms, systems and methods of use. The systems acquireand process data regarding current context of one or more users whois/are using or connected to a mobile communication device such as asmart phone or tablet—or any connected conventional or future devicessuch as augmented reality glasses or contact lenses, personalcommunication hubs, connected clothing, jewelry, shoes and oraccessories, or any embedded communication devices. Current context datais obtained using various sensors, sensor input, devices, imagerecognition or augmented reality residing in the connected device(s).Additional data from other devices, sensor systems, users and users'devices and/or physical locations may be added to define the environmentthat any user is in. Combining the contextual data on each connecteddevice with data gathered from other such sources in the environmentand/or proximity around the user(s) preferably define the completecontext that the user is in for the purpose of the present inventions.

Context is defined herein includes but is not limited to the activity,location, mental state, physical state, mode, direction and speed oftravel, level of engagement as well as the surrounding environment theuser is within, other person(s) who may be present or nearby, what othermobile devices are present or nearby, including other connected devices,equipment, vehicles, tools, computers, displays, point of sale systems,ticketing systems, inventory systems, video and audio capture sensors,ball(s), object and other tracking systems and other sensor system(s),devices and/or equipment that can communicate via a Wide Area Network orany other methodology directly to one or more end users or through aback-end cloud based network connected to a cellular or WiFi network orany future method of communication capable of such communication.

The current context data gathered on an end user's device is preferablyupdated via a wide area mobile network or local area network to acontextually intelligent server, preferably a “cloud” based server. Theserver matches the current context data provided by the device with theuser's past and historical data that has been collected previously via,preferably by other means and that is included in the cloud baseddatabase. The server communicates with the connected device(s)concerning a variety of contextually relevant information such as, forexample, feedback, experiences, recommendations, offers, coupons,advice, tactile feedback, and content such as visual and audiorepresentations, augmented reality, and other audio/visual displays.Preferably the information communicated between the device and theserver is useful and relevant to the user's current context and likelyto be useful to the user as the user enters a new context.

Optionally, identity information of each user is matched with pasthistorical data and the current contextual data of each user,respectively, in a customer relationship management system connected toa content server containing, for example, offers, videos, audio files,augmentations, 3D models, games, coupons or advice.

A cloud based customer relationship management system able tocommunicate with mobile devices delivers relevant information to theconnected device while the user is in a specific, determined context andupdates the information as the user moves to a next context, to a nextcontext and so on as the user continues to move from one context toanother context.

An image recognition systems such as Qualcomm's “Gimbal” and/orAugmented Reality systems such as Qualcomm's “Vuforia” preferably areused to capture and verify what product, image, ad, video, TV broadcast,billboard, poster and or display the end user is interacting with.

The physical distance between the image and the communication devicepreferably is used to capture the interaction preferably is used tocontrol the volume of audio feedback, strength of any tactile feedbackand the level of any other feedback delivered to the device.

Privacy settings on the device may be used to ensure that the usercontrols the nature and level of detail that is shared. End User LicenseAgreements (EULAs) define the use of the data collected. Detailscollected on the mobile device initially may reside on the device itselfand then be transmitted, preferably anonymously, or, optionally otherdata can be gathered and shared with the cloud-based system in a mannerwhere the device and/or user is identifiable, in accordance with localand international laws, rules and regulations regarding privacy laws andstorage and use of such information.

Mobile devices as intended for use in the present systems and processesinclude mobile phones and tablets. It is envisioned that in the futurenumerous products will include electronic, communication capabilities,such as glasses, eye contact lenses, personal communication hubs,watches, jewelry, accessories, shoes, hats, sports equipment,entertainment systems both visual, audio and tactile, apparel, devicesin electronic equipment that is placed in specific geographicallocations to enable communication with an end user, with a mobile deviceas well as to sensors and devices embodied in a human body.

AR and IR experiences are currently available on the mobile devicedisplay screens and are therefore adapted for use in the presentsystems. The present systems also have the capability to be used inconjunction with soon-to-be-released AR enabled glasses such as Google'sGlasses™ brand AR project and others, as well as other AR enabledcontact lenses that are in various stages of development. The presentsystems are also adaptable to work with other anticipated display systemsuch as reflective and interactive clothing and cloth materials, as wellas with displays in fixed location(s) in the environment as well asdisplays in various modes of transportation such as vehicles, planes andtrains.

Cloud storage is the concept of storing data and information on a serveror a combination of servers that resides in other geographicallocations. Numerous commercially available cloud storage solutions arecommercially available through companies such as Cisco and Amazon.

Big Data is defined as the collection of the data gathered in one ormore of numerous cloud based servers using one or many databases. Manysuch databases are commercially available, such as for example, Hoodad™brand and Mongo™ brand databases.

As each user moves around that user's surrounding context of the userwill change. During use of the present systems, information as definedabove will be collected and as the user moves, the corresponding,changing information will be delivered to the cloud based serves(s) sothat each user's context is always relevant and current, i.e., realtime.

As a user moves along through various environments, inducements andcontests can be presented and offered, at the right time and locationwith relevant prizes and offers, to the user. One such game can be a“scavenger hunt” or “treasure hunt” where the user is presented quizzesand puzzles that are relevant to the user's context and the prizesprovide value to the user based on past behavior and preferences of thatuser. As the user travels through the environment and solves thepuzzles, riddles and quizzes, new clues are revealed based on suchcriteria as location, whether an image has been found. IR or ARrecognizes when the user is at that specific location, and then anotherclue is unlocked and the user is presented an opportunity to solve thenext quiz, riddle or puzzle. This sequence repeats until the user endsat the final location or has found the right item that triggers theunlocking of the final prize. The final prize can be, for example, aticket to an event, back-stage passes, VIP treatment, coupon, exclusivecontent, physical or digital merchandise, games, video or audio content.

As the user continues through the physical environment, automaticupdates to the user's context and surroundings may be shared with othersbased on the user's permissions. Optionally these updates would be tothe user's social media status such as on FaceBook and Twitter forindividual users.

The system also includes use of AR tags in connection with geo-fenceareas by AR targeting by users and offering and displaying experienceswhen and relevant to a user's crossing a specific geo-fence. These canbe personalized and be made available to the right user based on theuser's data in the MCRM.

The system can also be used by law enforcement, military, firstresponders, natural emergency crews, security guards, border patrols,homeland defense, TSA and other personnel where a need to verify auser's health and safety is of interest. Contextual input such as audiodetection of a gun-shot can automatically be collected and reportedtogether with other vital information such as location, health, rate anddirection of travel that can also be immediately collected upon such anevent and reported to the cloud-based server where an alarm willautomatically alert to the situation. The user's context will then beimmediately available and appropriate supportive action can be taken asdirected by a superior or support group.

In other instances, the system can be used to detect what broadcast TVprogram, video, DVD, web delivered content and motion pictures such asmovies the user is currently watching. The system uses IR and ARtechnology to recognize certain images/frames/sections of the contentand matches this against the cloud based image recognition database.Once a match is made, the CRM system delivers relevant information tothe programming that the user can engage with such as updated stats,information on the participants in the content such as players and/oractors, associated and relevant games, fantasy games, ability topurchase items that are shown, ability to order products and food shown,and the delivery of these items are based on the user's currentlocation, context and past behavior. Secondary content that is relevantto the content optionally may be displayed on the user's device such asa different point of view (POV) or camera angle during a sporting event.

Other items such as clothing and/or markings/logos or hangtags attachedto the clothing can also act as the target for an AR/IR experience.Hangtags can trigger display videos of runway fashion shows, modelswearing the clothing, the designer describing the apparel, showingfeatures of the apparel and or the designer describing his motivationbehind the creation of the design. AR overlays can be superimposed onthe clothing directly, or can be added upon the person wearing theclothing. One such example may be a hat with a logo of an AmericanFootball team that triggers the display of the same team's helmetsuperimposed upon the hat and the wearer of the hat. All theseinteractions are preferably collected on the device and reported to thecloud based CRM.

Should the context include a retail location that is also part ofdesignated system in accordance with the present inventions, directcommunication between the retailer and their POS system can enable amobile payment solution using a convention payment process, such asQualcomm's “PAY” system. This system preferable is automaticallyactivated on the user's mobile device as the user enters the physicalcontext in which the system is active. The user then has the option toaccept the use of it, or decline. Additional offers and delivery ofcoupons and incentives to purchase the product with potential discountsare available should the retailer want these delivered.

Should there be other end-users in the proximity to an end-user and allhave at one time downloaded one of the applications that use the presentmethodology, an ad-hoc peer-to-peer communication can be establishedusing technologies such as Qualcomm's “AllJoyn” application. Onepreferred embodiment of the presently described inventive processes isuse of for visitors to locations, venues or events where there is a longline waiting to enter—such as at a theme park where people line up forroller coaster rides. In such a context, a location based game can betriggered and end users that have the solution on their devices can beconnected and start playing against each other. As an incentive, thewinner may get a front of the line pass if he wins, for example. Aspeople move away from the location, they will drop out of the game, andothers will be invited to join as they come into proximity. The gamecould be set up so that it can be played only by users that are in thespecific location or predetermined distance from the gate, whichcriterion is the context defined by the back-end MCRM system accordingto the theme part owner or operator.

As a person is moving around their surroundings, images can becomeroadmaps and markers. These can offer directional feedback to theend-user. They can be simple logos, arrows, or any images that the IR/ARtechnology recognizes as a directional sign as matched against the cloudbased target database. The image can indicate to one end user to go leftbased on that person's desired destination, the same image can indicateto another user to go right based on the person's desired destinationthat differs from the first person's destination.

Another embodiment of the presently disclosed processes is the creationof type set such as fonts with specific features, characters and signsthat are efficiently recognized by AR/IR and can quickly provide adynamic feedback, such as a “STOP” sign is recognized as an action to“STOP” because of its shape rather than the individual letters“S-T-O-P”.

Other posters, images, flyers can also be customized to appeal to theperson viewing them. This is achieved by using the cloud based imagerecognition database. Once an image is recognized, it will be matchedwith an associated AR or IR overlay. A selection of different options tobe presented and the AR/IR overlay displayed is based on the end-user'spreferences, gender, age and past behavior.

In sporting events where the players in the game use equipment, shoes orclothing with sensors that gather information such as direction and rateof speed, G-forces in turns or upon impact, video and audio feedback,and other sensors measuring the player's health and physical status,corresponding data can be collected, stored and processed in accordancewith the principles of the presently disclosed inventions. When an enduser is wearing a connected device, glasses, gloves, body panels or sitsin a connected chair with feedback display abilities such as impact,sound, shake, G-force simulation, the event can be replayed with acombination of these feedback aspects in addition to traditional contentdisplay such as broadcast TV.

In once such embodiment helmets worn in high impact sports with G-forcesensors and mouth strap sensors are adapted to include sensors. Should aplayer experience an impact with a G-Force that is deemed unhealthy inaccordance with predetermined criteria, a message can be sent throughthe system to a responsible person, such as the team trainer or a coachof the team. Should the mouth strap or other means to secure the helmetnot be correctly closed, locked or engaged, another message could alsobe conveyed, as yet another example of use.

Geo-specific messages are also enabled by the presently describedsystems and processes. One such example is in a stadiums or fields whereyouth play any such high impact sport, such as middle—and high schools,colleges and universities. By establishing geo-fences of theselocations, messages can be delivered as users enter the geo-fencesaround these locations. These could be messages such as messages fromprofessionals playing the sports to “play safe, buckle up your chinstrap” and that are delivered to users as they cross the geo-fence.

Additional sensors at a sporting event or on other equipment such as onboats and on cars, can gather other information from the event. Thisinformation includes, for example, ball trajectory, player positions,speed and direction of travel, and telemetry information from a race caror sail boat, both as it relates to the individual player, ball, car orboat but also how it relates to others in the event—competitors, teammates and official and other points or areas of interest. Theinformation gathered can be used to display the activity in an AR replayof the actual event—such as superimposing the trajectory of a baseballpitch in a 3D-AR environment, or displaying soccer game with real-timeball and player representation on a target in AR.

Another preferred embodiment of the presently described systems would befor business promotion and networking where a company's logo is theAR/IR target. Once recognized by a sensor, and processed, the systemwill bring up any of a variety of information, such as informationsuperimposed on the logo about the business, show content, displaycatalog of products, introduction by the CEO, 3D-graphic animation ofproducts and/or any aspect of the business that the business wantsdisplayed. Connections to LinkedIn, Twitter, Facebook and corporatewebsites can be displayed as interactive buttons. Once the end useractivates the AR experience, that user's corresponding information isretained in the cloud based CRM system so that the company can remain incontact with that user.

Corporations that have multiple departments, divisions, subsidiaries orother complex hierarchy and organizational charts including managementstructures, AR can be used to showcase the management hierarchy, withreporting and responsibilities superimposed on each level. The structurecan be displayed in 3-D and paths can be evaluated, highlighted andfocused on.

Another embodiment of the presently described systems and processes isuse of a company logo as an AR marker in the lobby or entry of acorporation. Once the AR experienced is triggered by the visitor to thecorporation, a video of the CEO will appear in AR welcoming the guest.The event is captured, reported to the cloud based CRM system and theninformation about the guest is sent to the corporation. They then haveinformation about their guests and can easily follow up with them in thefuture.

A company can also use the system to assist in making meetings betweenemployees and their customers, suppliers, clients and/or other entitiesthey interact with and have physical meetings with. Using the inventionback-end system, each participants of the meeting is in the samecontext. Proximity based communication between the participants can thenbe automatically engaged. Automatic recording of the event can beenabled on each participant's device and reported back to the MCRMsystem. Each participant can then at a later stage review theinformation gathered. The system also enables communication between theparticipants directly as well as with other devices in the same physicalor nearby locations such as displays, printers, AC/HVAC/Window Shadessystems that control or influence the environment. As the participantsleave, and others enter, the context of each user changes, but thephysical contextual bubble of influence generated for the specificmeeting remains the same until the MCRM system has been set up to endthe event.

Another embodiment of the solution is to create “360 degree” fanexperiences. In this example a fan of a sporting team purchases a ticketand downloads the team's mobile application. This application featuresan AR or IR component that creates an AR or game experience superimposedon the ticket as part of the use of the presently describe ecosystemsystems and methods. This interaction is captured by the cloud basedMCRM system verifying that this is a fan that is coming to a game, aswell as the identity of the device and the location of the user. Oncethe user is in transit on the way to the stadium, the mapping feature ofthe application detect that the user is coming close to the stadium. Ifthere is a game that day it is likely that he is heading to the game. Arelevant offer to a participating retailer close to the stadium can bepresented as the user approaches. Once the user crosses a predeterminedgeo-fence around the stadium, as selected by the operator of the MCRMsystem, it is likely that the user is going to the game, and offersrelated to the game and venue such as coupons for team merchandise orconcession stands offers can be presented to the user's mobile device,and if applicable, past historical data will be combined to ensurerelevance and high desirability of the offer to the user. An offer toenter the stadium via a VIP entrance can be presented. While in thestadium, engaging and entertaining games can be presented such asproximity based games can be presented, proximity based AR replays oflast week's game can be displayed at specific locations. These games andexperiences can only be played while in the stadium and during certaintime slots such as before game starts and in breaks in game play. TheMCRM system can also communicate with the stadium systems such as bigscreen displays such as “Jumbo Tron” displays. The results of the gamesplayed by the fans can be displayed. Exclusive content from the gamesuch as alternative camera angles or different point of view video feedssuch as from a driver's seat or a referee's cap that has a video cameraembedded can be presented to the fans in the stands. As the fan leavesthe stadium, the fan's connected device will cross the establishedgeo-fence and a “thank you for coming” message can be displayed. Thiscan include an interactive button with an offer and a call to actionsuch as “come back Thursday night—buy now and get 2 tickets for theprice of one”, for example. Once the fan has returned home, content suchas exclusive high lights from the game can be presented, with anotheroffer to purchase tickets for a future game. The process then will berepeated based on the new tickets. Frequent visitors can be rewarded andloyalty programs can be created, all managed by the MCRM.

While attending a sporting event in the stadium, another game can betriggered as an AR feature where the user takes a photo with a virtualAR representation of one of the players on the team. By taking thephoto, the MCRM system captures the interaction.

A suggestion to share this on social media such as FaceBook with friendsto win a prize is yet another example of use. Those who share with apre-determined number of friends will win a prize, for example. The onewho shares with the most will win the grand prize. Each photo couldinclude a suggestion to download the team's application with apossibility to win a prize if the recipient of the photo does so. Theback-end MCRM system is programmed to capture each such interaction andis capable of automatically delivering the rewards based onpre-determined action levels.

As the user who is in the stadium and has had the application downloadedapproaches a concession stand, the ad hoc proximity peer-to-peer systemwill create a connection with the concessions stand's mobile device/andor connected point of sale (POS) system. The menu from the concessionsstand will be displayed on the user's mobile device. The user can selectthe menu items and the order will be transmitted to the concessionsstand's mobile device and/or connected POS system. Once the order hasbeen placed, a cloud based mobile payment system such as Qualcomm, Inc.'s “PAY” solution or near field communication (NFC) payment solution suchas Google “Wallet” can be used to complete the transaction. When “PAY”is used, the users purchase is captured and retained in the MCRM system.

In another embodiment, a “360 degree retail experience” can be created,managed and operated using the present platform. In this instance, theentry point to the experience is the retailer's advertisements such as aprinted “circular” advertisement inserted in newspapers or traditionalads in newspapers and magazines. Using the retailer's mobile app thathas been enhanced with an embodiment of the presently disclosed systems,AR and or IR is used to register the user's interaction with the ad orspecific sections of the ad. AR experiences demonstrating the productfor sale, or content that tells a story about the product can bedisplayed and superimposed on the ad. Additional information and orcoupons can also be displayed. As the user approaches the retailer whohas used the MCRM to create a geo-fence around its physical retaillocations, offers and coupons that are relevant to the user will bepresented as the user enters the geo-fenced area and is ready by thetime the user enters the retail establishment.

If a story is displayed as content in AR, the beginning of the storypreferably will be superimposed on the ad. To learn what happens next,the user has to go into the retailer's physical retail location andlocate another AR trigger. Once this has been located, the next chapterof the story will be revealed, and a coupon or other reward will bepresented. Once the user has purchased the item, taken it home andopened it, and another chapter will be revealed and the story concludes.The story line content was created once and uploaded to the cloud basedcontent database for playback on the relevant user's mobile devices.

The present systems and methods can also be used to record and store auser's wish list of products. The user can “tag” an item the user wouldlike, upload it to the MCRM, and store who else the user want to informof the desire to receive the product. Once this other user enters thelocation where the product was originally tagged, or another physicallocation that has the same product, the second user will receive analert that there is a product that has been tagged and put on a wishlist by the first user. If there is a connected POS system in thelocation, this will also be alerted to the presence of the 2nd user.

AR can also be used in the retail location to showcase what's inside thepackaging. Once tagged by the user as desirable, the MCRM will recordthis and remind the user if the user delays the purchase to anotherlater time.

In another embodiment, an advertiser and/or advertising agency canmeasure the effectiveness of its print and advertising campaigns. Amobile app with the AR/IR capability and the MCRM back-end systemenabled, the advertiser can measure length and frequency of adinteractions. If call-to-actions are presented in the ad or in the appitself, the effectiveness of these will be captured by the MCRM system.If a call-to-action is for the user to enter a specific physicallocation such as a bar, restaurant, retailer, venue, mall, cinema,sports venue, expo, conference center, business establishment or hotel,the MCRM system will map the user and verify when the user enters thelocation. This is called Contextually Driven Proof of Presence. If thereare items displayed for sale, a “buy now” button can be integrated andthe sale will be tracked and verify by the back-end system. If there arepromotions for an event, a movie, or concert, the date, location of thenearest theater, show times and “buy ticket now” including directions tothe theatre can be displayed.

There are currently a few MCRM systems commercially available such asParallel6's CaptiveReach™, and the inventions described herein are notdependent upon a specific MCRM solution.

Another use case for the systems is for the use of crowd control. Inlocations such as a theme park, the treasure hunt game can be created sothat it drives the users participating to a specific location at aspecific time. For instance, it can end up in the middle of the park at5 μm, and as the user is walking towards the exits, offers and couponsfor discounts in restaurants will be presented as the user is walkingpast them inside the park. The likelihood of the visitor leaving thepark at such a time is lower and it's beneficial to the park owner tohave their visitors dine inside the park rather than after exiting itsgates and dining at another non related establishment.

While AR can be simple video overlays, the presently described systemsinclude AR interactive games. One such embodiment is an applicationwhere an interactive penalty kick soccer game is displayed in “half AR”mode which is the definition of an experience where half the viewingdisplay of the mobile device is displaying real-life surroundings, whilethe other portion is displaying a virtual world where the game isplayed. This can therefore effectively be played during breaks whenattending a soccer tournament, with the arranger of the tournamentcontrolling and presenting branded targets that trigger the experience.The game can be branded with specific team jerseys and uniforms worn bythe avatars in the game. The game can also serve as a tool for the teamfor fund raising efforts. This can be achieved by offering sponsors theability to purchase ad space in the game. In addition, the interactionwith the game can be collected by the MCRM system and when the user isin proximity at certain times to other sponsors of the team, offers andcoupons can be delivered to the user. One such instance can be a localpizza parlor that sponsors a youth soccer team. After the game as theteam players and their families are leaving the game and in proximity ofthe aforementioned pizza parlor, an offer to come in and save on thepurchase of a family meal may be delivered. The team will receive aportion of the pizza parlors cost of using the system.

Using the AR as a way to focus the user's attention on a specificproducts is achieved by creating a “ghost” image of the object to befound with a washed out and transparent image of the target to be found,as is conventionally known. Once the image is found and “locked” on, anUI animation can simulate the “lock-on” in the UI of the device.

When using AR in real life, the ability to maintain the AR target inview at all times is not always feasible and the AR experience willtypically lose traction and the overlay will no longer be displayed. Thepresently described system includes the ability for an AR overlay to betriggered and started by the target, but if the target has been noted inthe database as one that will be allowed to break the visual connectionneeded to be superimposed, it enables the AR experience to be portedonto the viewfinder or display on the device and continue to bedisplayed.

By using AR targets as posters in real life locations that arerecognized as targets in the cloud based target database when it is inthe view finder of a mobile device and that target is recognized, theuser's mobile device is recognized as being interacting with the targetby the MCRM system. Using the knowledge of the user from the datacollected in the MCRM and the historical data inserted and included inthe MCRM, the AR overlays are selected from another database. The ARoverlays in this database have been specifically created based on theuser's parameters such as gender, age, interests, ethnicity, pastbehavior, preferences and other pre-determined aspects. The AR displayedon the user's device will be tailored to the user's known parameters topresent a personalized experience. The same poster will thereforepresent a different experience for a single young male user as to anolder mother of two. The effectiveness of the differentiation can bemeasured and captured by the length of the interaction. If there isproduct displayed for sale, a “buy now” button can be integrated and thesale will be tracked and verify by the back-end system. If there is apromotion for a movie, movie schedules for the nearest movie theater canbe displayed, directions to get there and “buy ticket now” featureintegrated.

Another use of the systems is the AR enhancements of comic books andmagazines, as well as catalogs used to present and sell merchandisewhere pages are divided in sections in a similar fashion to the comicbook layouts. An AR avatar of one of the figures in the comic book or arepresentative or model featured in the catalog can emerge out of thebook or catalog. This avatar can become interactive and guide and leadthe user reading the book/catalog as the user progresses through thebook. The avatar in the comic book can add games, show hints, additionalcontent and other information that is relevant to the story line. Theavatar in the catalog can present specific products, remember what thereader likes, collect a wish list, present related products, suggestgoing to a retail store, suggest and enable the user to purchaseproducts instantly and present other relevant games and or content andor information relating to the products shown.

The AR games can also be triggered by targets such as collectible cards.These can be displaying avatars of known athletes such as NFL players orfantasy figures from games, TV series or movies. The characteristics ofeach player/character can be displayed on the card and incorporated inthe AR display of the character/player. This can be dynamically updatedsuch as in the case of the NFL player where real life stats can beupdated and become a part of a fantasy league. Strengths, weaknesses,mode of battle, agility and other characteristics of fantasy figure canalso be displayed, both static or dynamically. When more than one cardis present, the AR experiences will be collected and recorded by theMCRM system. Dynamic interactions can be created and used to control theAR displays of the avatars based on this information. When more than oneuser is present, any ad-hoc peer-to-peer communication solution such asQualcomm's All-Joyn or Blue Tooth can enable game-play between the usersbased on the cards present. All of the user's devices will communicatewith the back-end MCRM system that will capture each interaction both onthe physical cards, as well as interaction with the AR dynamicallycontrolled avatars.

The AR experiences can also be triggered by targets that users create ontheir own. By having a target with known characteristics on the bordersand or corners, the center of the target can be blank. One embodiment ofthis feature is the ability to enable a child to draw an image of thechild's favorite carton or TV character that is associated with thetarget and the target is a piece of paper with borders and perhaps thename of the character. The parent of the child can explain to the childthat the target is specific for a specific character such as SesameStreet's Elmo, for instance. The child then draws the child's renderingof Elmo in the center of the sheet of paper. Once the target isrecognized by the borders and corners, an AR 3-D model of the characteris displayed on the connected mobile device.

Another embodiment of the solution is for use in education. Text booksand other printed material can trigger puzzles and problems to besolved. For example, these can vary from placing AR cubes with “A”, “B”and “C” in the right order, or performing a virtual AR dissection of afrog in 3-D where mistakes in the sequence are noted. Once performedcorrectly, this is reported to the MCRM system. Another subsequentproblem or puzzle to solve is presented on the same target. Only uponsuccessfully completing this, the next puzzle or problem to be solvedwill be presented. The MCRM system will track and verify each step ofthe learning process. Results and progress can be gathered and presentedto a teacher's mobile and/or connected device. Individual results andprogress can be reported to the caregiver and/or educator of the childor student.

Another embodiment of the solution is the use of proximity based gamesin venues and locations such as bars, nightclubs, sporting venues andplaces where users would come to mingle and interact socially. Thelocation based proximity games that are powered by the ad-hocpeer-to-peer communications will be triggered by a target that is commonfor all the players such as a restaurant menu, logo, beer coaster orother specific image. Once triggered and the users are in the samelocation, the game will start. Each player will see what the otherplayer(s) is doing as displayed on their own device. A simple game suchas a virtual “musical chairs” can be played where each player has toperform a function within a certain time and virtually flick to theplayer sitting next to the user before the music ends. Harder tasks andshorter time allotment will make the game increasingly hard to play.Once a winner emerges the time and other scores are reported to thecloud based MCRM system. Leagues, loyalty programs, leader boards andgames across location, cities, countries and continents can be enabled.In-game and in-app sponsorships and ability to purchase digital power-upin-side the game can be enabled. The game can be specifically brandedfor specific locations, brands, teams or countries. The game can bespecifically tied in to a location or specific locations. In-gameexperiences, scores and results can be shared via social networks. TheMCRM system will capture these and reward those that meet the licenseeof the game's criteria.

In another embodiment of the solution, an offer for a free drink ispresented to male patrons that are inside a specific bar that uses theMCRM system to present the offers directly to patrons inside the bar.However, in this instance, for example the offer for the free drinkcould be limited to a female patron and a short time duration. The barowner has thereby encouraged their male patrons to engage with femalepatrons with a limited time and specific offer to create socialinteractions between the patrons of the bar.

The presently describes systems and can also be used with instructionmanuals, how-to guides and cooking books. The present systems add theability to present step-by-step instructions then verify that they userhas correctly followed the steps. It tracks and verifies the user'sinteraction with the printed material and collects additional contextualinformation. Based on the knowledge that the user is engaged in anactivity such as assembling a piece of furniture, repair a car or cook ameal, combined with the user's location and past preferences andbehavior, relevant and helpful additional information can be presented.In addition, the awareness of the user's context will enable thedelivery of highly relevant offers, tips and other content. If purchaseof a related product is considered to be worthwhile, then a “buy now”button can be integrated and the sale will be tracked and verified bythe back-end system.

Interactive, personalized and dynamic content can also be presentedusing the present inventions. The content is customized andpersonalized. One such embodiment is the display of weather forecasts ina newspaper. The weather section of the newspaper would be the targetfor the AR application for the newspaper with the system. Instead ofpresenting the static national weather that the printed versionpresents, hyper-local weather forecast is presented based, on the user'scurrent location. The MCRM system will gather the information and matchwith dynamic weather forecast data from any of the many commerciallyavailable weather forecasting services such as from www.weather.com.

Another embodiment of the present inventions is the creation of dynamicupdatable printed magazines and publications. Sections, features and adscan be designed and created so that they depict images that representthe first frame of a video. Once the phone is held over the image, themobile phone's camera recognizes the image and the video starts. Thiscould be from a fashion show, from a movie, DVD, video tape, or anyother motion capture display methods. The interactions are captured intothe MCRM. Dynamic and updated content can be created and displayed atlater time. The printed material is therefore becoming the portal tocontinued updated material.

Printed comic books and graphic books can also be enhanced with thesolution. Using the AR and IR capabilities to recognize sections and orpanels within the book, additional AR and or videos or animations can besuperimposed on the selected images. These can be served up in a randomfashion or sequences can be controlled by the MCRM system. If the comicbook is associated with a video, TV program and or motion picture, videoclips can be superimposed. If there is content for sale, a “buy now”button can be integrated and the sale will be tracked and verify by theback-end system. If there is a motion picture, movie schedules for thetheater nearest the user can be displayed.

Catalogs of products for sale can also be enhanced and tied into aretailers commerce channels. Initially an AR avatar can emerge andprovide guidance and advice, showcase features and make recommendationsto the user. Demographic and past contextual data can be used to makerecommendations. The user's behavior during the review of the catalogwill be collected and recorded. If the user spends a longer time on aspecific product, the avatar can be programmed to focus more on thiscategory of products as the user goes through the catalog. The user cancreate a “wish list” of the products the user likes and wants. The MCRMsystem will record this and remember it. If the user goes to one of thecatalog owner's retail locations, the list will be presented. If theretailer has a POS system that is connected with the cloud, the MCRM canpush the wish list to the store POS where the sales staff can be alertedof the user's presence and desired items. The MCRM system will alsoallow the user to share the “wish list” with friends and family—bothdirectly from the app as well as through the user's social medianetwork. The concept of a 360 degree retail experience can beaccomplished by having the avatar present the beginning of a story thatwill be completed once the user enters one of the retailer's physicalretail locations. Completing the story will be accomplished bypurchasing a product. A complete mobile payment solution such asQualcomm's PAY can be used in the retail store to complete thetransaction without the customer needing to approach the check-outregister.

Another embodiment uses the AR targets as communication channels. Imagesplaced on items such as, but not limited to, apparel, photos, ads,logos, products, packaging, posters will become a distribution channelof digital content that can be superimposed onto the target. Thisinformation can be content such as news, updates, announcements,entertainment, encouragement, supportive messages, reminders and alertsthat are relevant to the product that has the image on it or is depictedin the image or is associated with the image. Personalization of thecontent can be created by the MCRM back-end system that drives anddelivers the messaging in the AR experience from the AR database,combined with the user's current and past contexts and past behavior,preferences and behavior.

Traditional maps can also serve as dynamic AR targets. One such instanceis a tourist map such as those frequently given away at hotels andairports at tourist destinations. Once the target is recognized by theuser and the user is no longer in the user's home town, the MCRM systemwill recognize that the user is in a new location. AR experiences willbe presented on the map that are relevant to the users past behavior,known likes and desires and most likely predicted wishes. The AR overlayof the same map will therefore be customized and personalized to the enduser's past preferences—and will differ from user to user. The contentdatabase will store the information and will be connected with the map,the user's mobile device and controlled by the MCRM. For instance,consider a user who lives in New Your City, arrives at San Diego airportand uses his mobile device on one of the tourist maps available at theairport. The MCRM system records the new location and recognizes thetourist map and then concludes that the user is most likely on avacation. The MCRM system knows the user is frequenting steak houses inNYC and will present offers and directions to participating steakrestaurants in San Diego and that may also be super-imposed on the map.As the user moves around in San Diego, offers that are relevant to hispast behavior, preferences, desires and history are presented at theright time and locations.

Another use example is creation of interactive, dynamic and personalizedAR experiences superimposed on the printed map. This use can featureinformation, offers, coupons and details about locations and retailersand it can provide directions to locations and present highlights on theroute chosen.

The AR map can then be ported onto the device and remain on the devicewith the user's location updated as the user proceeds along the selectedroute.

Another use example is creation of automatic check-in and registrationfor users that enter a specific location where a payment, registrationor notification of the user's presence is required. One such use may bethe registration of guests in a hotel lobby, participants in an expo orattendees at a concert. Here the user would have registered prior toarriving, and optionally previously could have experienced an ARenhancement associated with the booking of the hotel, expo or event. TheMCRM system registers this booking, and when the user crosses thegeo-fence established by the location, the MCRM registers this and sendsa message to the order, booking, and registration and/or ticket officesystem at the physical location, thus completing the check-in processautomatically. The user could additionally be presented with an optionto do this automatically upon the initial AR engagement, or reminded todo so or have to do so manually when crossing the geo-fence.

In certain physical locations where GPS signals are weak—such asindoors, event centers, casinos and in large shopping malls—the presentsystems enable AR/IR targets to become virtual guides and sign posts. Byrecognizing targets that are placed in a known physical location, theMCRM system will capture the target and match it against the targetdatabase. Since the physical location of the target is known, the user'slocation is also known because the user needs to be in front of thetarget for the AR/IR to recognize it. The MCRM system can then sendinformation about other known physical places in proximity to thecurrent AR/IR target. Information such as directions, landmarks,distance, features of the location and offers from the location can bedisplayed. These will be superimposed on a digital map and would beunlike the traditional AR experiences, where the user holds the phone upagainst the horizon. These conventional systems are typically dependenton GPS coordinates of the user and the AR overlays and do not workindoors. The present system uses a combination of the user's knownlocation, the MCRM database of known and defined locations, the MCRM'sknowledge of AR/IR target's locations, and if a known number of theseAR/IR targets are recognized as dynamic directional markers, directionalarrows can appear as the user moves through the environment guiding theuser to the desired end location. Other use cases for this can be inmilitary environments where the markers can become hidden directionaland warning signs or in disaster areas where vision and otherdirectional signs and senses are hampered.

The presently described systems and processes also enable creation andcommercial delivery of widely adaptable human body area networks. Acentral hub provides communication with the cloud based MCRM system. Itcollects data from numerous sensors, some embedded within the body,others worn directly on the body, others worn in apparel, jewelry,accessories, hats and shoes, others placed on electronic devices, othersplaced inside vehicles of transportation, others placed in theenvironment. Data collected is based on the user's context, collected bythe hub and transmitted to the MCRM system. Data from other sources thatis relevant to the user will also be collected by the “Big Data”database of the MCRM system. Based on the user's current contextincluding health and mental state, location, activities, desires, pastbehavior and preferences, information will be sent to the hub. The hubwill then send the information in the appropriate format to variousfeedback devices such as glasses and contact lenses that can displayimages and AR, audio devices that can play back sounds, tactile feedbacksensors that can provide feelings of touching, bumpers that can be wornto simulate impacts in sports events, clothing items such as sleevesthat can display images, and accessories that can display images.

The communication hub functions as a thin client connected with thecloud for storage. The battery power on the device is used for computingpower and low power communication with sensor input and display output.Storage information is preferably retained in the cloud database.

As the user moves into an environment where there are display devicesthat are connected, a connection directly between the hub and thedevices may be established by a variety of methods, and information fromthe hub and the cloud based databases can be displayed on the displays.

The solution also enables digital signs to display relevant informationas a user moves through an environment. The display sign is connectedwith the MCRM system in the cloud. Once a user's context bubble entersthe “sphere of influence” of the digital sign, the user's mobile devicewill recognize the event. The MCRM system will recognize the event fromboth the digital display as well as the user. The MCRM will send anappropriate message to be displayed on the display based on the userspast preferences, current state of mind, the user's likely nextdestination as well as any recorded activities in the future. Thedisplay will be connected to a content database in the cloud that iscontrolled by the MCRM system. This enables the display to present thecontent that is relevant to the user as the user is in proximity to thedisplay.

Another embodiment of the solution is the ability to serve as acontextual browser. By combining AR and IR for visual input and replayof search results, the system's ability to capture contextual data fromnumerous users simultaneously enables the system to create a search forthe user for a context that meets the user's desires. Once such use caseis where the user is in a new town Saturday night and wants to havedinner in a place that presents coupons and offers, has loud music and ahigh density of people of the opposite gender within an age groupsimilar to the user. A traditional web search will list locations closeby, but without real-time contextual data. When a restaurant is usingthe invention, the number of patrons using the system, their data andpreferences and other data that they have opted in to share are capturedby the MCRM system. A real time contextual representation of the stateof that specific location is thereby gathered. If that matches theinitial user's desired context, and it matches the user's pastpreferences and desires, that location will be presented as arecommendation by the system to the initial user.

Another embodiment is for monitoring, verifying and tracking the user'shealth, the user's benefits from consuming specific health relatedproducts, medications and treatments, and the user's behavior as itrelates to the user's health. By combining information about the user'shealth condition as collected by a variety of sensors worn, placed uponskin and/or embodied into the user's body, the information is becollected by the user's device, recorded and communicated to a separateHIPA compliant database. This database features security measures andthe user's device only collects health sensitive data that it can verifyas the user's specific personal data, and biometric systems are used toverify. Numerous database systems such as these are commerciallyavailable as are numerous biometric personal recognition systems withthe present systems not being dependent upon any specific conventionalsystem, application or process. The database that collects the medicaldata is linked with the MCRM system, where the user's identity iscombined, not the medical data. The user's context is recorded by theMCRM system, and information that is relevant to the user's health isreported to the medical database. This can include contextual data suchas distance walked during a day, frequency, number of and lengths oftime spent at fitness and spa locations, activities in sport, andstationary time at the office and at home in front to TV, video games orcomputer. Reminders, encouragements and advice can be sent to the userif user's behavior is not healthy, for instance as a user is approachinga fitness center that uses the MCRM system of the invention, an offer tocome in for a free class may be presented with a reminder to the userthat it has been X number of days since the user last exercised.

The solution can also track and verify the benefits of medicine,treatments and products that have claimed health benefits. Theconsumption of the products and medicine can be captured and verified byAR capture of the products as the user is about to consume it. This canbe achieved by having the outside label of the packaging as one ARtarget that triggers the system to initiate data collection from varioussensors monitoring, capturing and recording the users health conditionsand stats. Once the user opens the packaging, there may be another ARtrigger inside the cap of the bottle containing the product, and thistriggers another AR experience that also is captured and used asverification that the product has been consumed. Rewards, reminders andencouragements can be displayed on the product to entice the user toactivate the recording of the use. When the user wears sensor systemsthat measure health conditions, the AR can trigger the sensors to recordthe user's condition before consuming the product, then at certaintime(s) thereafter. Activity levels and heath conditions will berecorded and stored in the approved medical database.

A commercially available product with perceived but unproven healthbenefits can also be enhanced with the present system. The user'sconsumption of the product and the verification thereof, the dailyrecording of activities, the reminders and encouragement to live ahealthy life style can all be presented as an overall solution by theproducer and owner of the commercial product. In such an instance, therewould not be any claims to the product's benefits other than formarketing purposes and only in compliance with local and internationalregulations, laws and policies related to proclaimed health benefits. Byusing a secure database that is in compliance, any data that is personaland can be construed as needing to be in compliance with medical healthrecord policies will be ensured as this data is retained and storedseparately from the MCRM.

Another embodiment is the use for nutritional information on food andliquid products. For each participating product, their label/packagingwill be stored in the content database. When the user is displaying anAR experience on the product, the nutritional information is overlaidupon the product. The user then is prompted to answer whether theproduct is to be consumed or has been. If consumed, the calories, thevitamins and nutritional elements of the products and other relevantcharacteristics that have been entered into the database associated withthe product will be recorded in the user's medical database. A summaryof the daily consumption will be presented to the MCRM and will bereported back to the user at the right time and location. This can beused by the system to verify, record, remind user to take medications orwarn users against over dosage.

The ability to record a user's daily context and replay it at a latertime is another embodiment of use of the present systems. Simpledisplays of the summary of a user's day can be replayed and presented atthe end of the day, or when certain pre-determined levels are met, overor under the amount. For instance the system can verify how far the userhas walked in a day, and if the level is below the user's desired dailywalking distance, it can send the user a reminder after dinner to talk awalk at the distance that will bring the user's daily walking distanceto the desired level. Suggested routes may also be presented.

In the future, it is envisioned that the user can display an avatar ofthe user in a digital format and superimpose this avatar into ARexperiences that the user replays or shares with others real time or forreplay at a later time. Other's avatars that are in the same contextualenvironment can also be superimposed into the virtual experience. TheMCRM system will capture the initial context; interact with the ARdatabases and the replay systems to recreate the context experience withthe user's avatar inserted. Data collected by the user's mobile hubestablishing the user's complete context will be collected and used tore-create the contextual situation of the user and display on theavatar.

A user's current context can also be defined by the user's location.There are numerous means to define and record this, and the solution isnot dependent upon any specific method.

One specific embodiment of the solution can create hyper sensitivelocations of the user. By having a controlled environment that has beenpre-recorded and defined by the MCRM system, numerous specific Wi-Fibeacons in highly accurate locations, can be used to precisely locatethe user in that specific location and context.

One such implementation can be used to determine if a user wearing aconnected device is sitting in the passenger seat or driver seat of avehicle. The vehicle will have predetermined beacons installed eitherfrom the factory or installed at a later time. For instance, thespecific vehicle Wi-Fi signature will be recorded and retained by theMCRM. Once the user enters the vehicle environment, the mobile devicewill receive the foot print of the signals and make a match to locatethe device. If the user is in the driver seat, certain functions can beenabled, other disabled. For instance, if the device is AR glasses orcontact lenses, the display can present the driver directional and otherinformation in an overlay fashion. Likewise it will disable tactileinput such as keyboard texting, and replace it with voice input only.Furthermore, if the user is a passenger in the car, tactile informationwill be accepted as input and vehicle information disabled.

As the user is moving about in the physical world, the user's contextwill constantly change. The user can use the MCRM to specifically recordcertain aspects of the context and share it with others. One suchembodiment is to place a tag on the specific location a user is in, andthen share this location with others either directly to one otherperson, or to that person through social media, or share with a group ofothers. This can be used for social interactions, but also for military,special forces, law enforcement, disaster relief workers where there isa benefit to let others know your location.

Interiors, other surroundings and environments can be enhanced with ARoverlays. These can be educational in locations such as museums, or atentertainment in places such as movie theaters, venues, bars andrestaurants, informative in hotels, casinos and shopping malls, orencouraging, relaxing or stimulating in private homes. The AR overlayswill adapt to the user entering into the environment and enhance thecontext that the user is in. The AR overlays will also change based ontime of day and the objective of the AR experience. In a retailenvironment, the AR enhancements can be used to encourage sales ofspecific products. In museums the AR can highlight specific art pieceswith information that is not interfering with the physical presentationof the piece of art.

Another embodiment is use for live performances such as concerts, circusacts and theater performances. Aspects within the performance such asprops, stands, displays, mannequins, statues, dolls can serve as ARmarkers and pre-recorded acts can be superimposed on them during theperformance. This can be AR overlay that is common and seen the same wayby all in the audience, or it can be customized to be randomly displayedor specifically presented based on the end user's demographics such asage and or gender. The theater or venue that the live performance is incan also be enhanced with AR superimposed content that can be same forall the users in the audience, randomly presented or customized forspecific audience members.

Certain locations can be enhanced with contextually specific features.One such example is an AR experience that enables and enhances bettingand wagering within horse race tracks or casinos. Offers that arespecific to such locations can be automatically presented to users thatare in such a context and have a desire to receive them.

Within other venues, behavior that is beneficial to the venue can berewarded. Loyalty programs, special front of the line entrance and otherVIP treatment can be awarded to users that are in the context andpromote to their social network that they are there. Should theirfriends respond in a way that is beneficial to the venue owner, suchuser will receive the benefits, status and loyalty rewards.

As the user moves through the physical world, the context will berecorded by a variety of sensors. In the future, it is anticipated thatthe complete environment can be recorded including sounds, ambientnoise, visual and audio experiences, smell, temperature, light levels,wind and breeze, and that this complete context can be recorded. Oncethe user enters an environment where there are control systems that cancontrol the environment, this information can be used to control andre-create the context in this new environment.

Users that have large social networks and have become known to be socialinfluencers will receive a specific status. Upon entering a contextlocation where the owner of the location desires to reward socialinfluencers, the MCRM will automatically reward and entice the user toshare and promote the experience. The MCRM system will reward the userbased on success or number of messages, invites, blogs, links, “likes”etc. and the user may in some cases receive free merchandise andproducts.

The presently described system accept current mobile device user controlinterfaces without depending on any specific methodology. In the future,controlling the devices and their interface will be achieved withgestures of arms and fingers as captured by a variety of gesturerecognitions systems. Other methods include body tracking and eyetracking as means for controlling the communication devices.

The present inventions include a simple way to control thecommunications a user would like to receive at any time. One suchfeatures is that by placing the device with the display facing down, thedevice will recognize this as an input and command to not be contactedexcept in emergencies. This can be used when the user is in a meetingand does not want to be disturbed with messages, offers or calls. Thisdesired state of “do not disturb” will be retained by the MCRM andmessages and offers will be placed on hold until the user changes themode of the phone.

The ability to record video in 3D and display video in 3D on mobiledevices, AR glasses and contact lenses is also be enabled and supportedby the solution. By capturing an event or person with a 3D camera, thecontext associated with that recording will also be captured. Oncereplayed by the person making the recording or anyone else the userwants to share this with, the 3D replay will also replay the context anddisplay this in the manner a connected context replay system will beable to display and recreate the context.

The application is a complete system and communication solution. It isenvisioned that it will be used by numerous entities under their brandnames in a variety of business arrangements such as “Platform as aService ” (Paas), “Software as a Service” (Saas) and or “Mobile as aService ” (Maas). It can also power a new ways for institutions, unions,sports leagues, schools, political parties and clubs to drivesponsorships and fund raising. By creating a simple AR game orexperience that can be customized for the entity, the entity can haveits teams, supporters and or members sell and promote the sale of theapplication through the variety of means an app can be sold such asApple's “App Store” and Google's “Play”. A portion of the sale will beallocated to the entity. In-application sponsorships, advertisements andsales of digital goods will add revenues to be shared by the creator,publisher and the entity. In addition, the fund raising entity'smembership can bring in local retailers, restaurants and merchants thatwant to present offers and coupons to the entity's membership creating aloyalty program between the location's patron's and customers, thepatron and the fundraising entity.

The way to commercialize the system can be a combination of pay forservice levels. These can include the cost of developing the front endapplications on a mobile device, then and additional fee to connectthese applications with the MCRM system. Whenever a message,information, offer, coupon or advertisement is sent from the MCRMsystem, a fee will be collected from the company sending the message. Ifthe message is opened and viewed by the user, the system will recognizeand record this and another fee will be collected by the sender of themessage. If there is a call to action embedded in the message and thisis acted upon, the system with recognize and record this and another feeis charged to the sender of the message. If a user enters a locationthat is owed by a client user of solution as a result of the clients useof the system, this is consider a proof of presence event and the MCRMsystem will recognize and record this event, and the client will becharged another fee. Should the communication presented and initiated bythe system result in a sale, a portion of the sale will be paid to theoperator and owner of the solution. The management of and use of thesystem can also be charged as a fee to the clients using the system tocommunicate with their end users. The use of the system in specificgeographical areas or within specific vertical markets can also bearranged to be a “master license” arrangement with associated usagelength of time compensation. It is envisioned that end user will not beasked to pay for any of the services, solutions or offers they receive.

The presently described systems can be used by organizations that havesubsidiaries, franchisees or sports leagues that have teams under theirumbrella management. In the case of a franchise operation, the back-endsystem can be programmed so that the franchisor controls the completesolution. They can drive and manage national or global marketing effortsand promotions from their master dashboard control system of the MCRM.These efforts can still be based on locations and context awareness andcan be specifically targeted to user segments or locations, or sent tothe whole user database that is under the corporation. Local franchiseeowners and operators may have access to a lower level set of datadisplayed on their own unique database dashboard interface. This will belimited in its ability to deliver messages to their specificgeographical territory they have approval to communicate within. Themaster franchise operator will be able to monitor, assist, encourage andstop messages and content delivered by the individual franchiseeoperators. One such use case is a national fast food chain that will usethe master dashboard to access every person who has downloaded and usedthe application on their mobile devices. Nationwide advertisements willbe displayed and managed by the franchisor. Local marketing campaignsthat are specific for a region such as warm coffee to owners in Idahoduring winter and milkshakes to operators in Florida during the sametime frame can thus be operated, performed and managed by each franchiseoperator with oversight by the franchisor company. Another use case is anational or global sports league that has teams in specific locations.The master control of the back-end system with access to all the membersregardless of team affiliation or location lie with the sport leagueowners. The owners can thereby send nationwide or global messages,sponsored content and ads to the whole universe of fans, regardless oftheir physical geographical location and or team affiliation. Individualteams will have access to limited set of functions of the MCRM systemvia their dashboard interfaces that are restricted to their geographicalarea, with the ability to reach their proven fans that may reside ortravel outside their geographical area.

The system can be used to drive sales by retailers, manufacturers ofproducts, creators, enablers and owners of content, owners and operatorsof venues, hotels, malls, offices, restaurants, night clubs, sport teamsand leagues. It can be used to create loyalty for sport clubs, musicaland theatrical actors and performers as well as their management anddistributors. It can be used to entertain by broadcasters of TVprograms, video and movie producers. It can be used by advertisers,advertising agencies and marketing companies. It can be used bypublishers of printed materials such as magazines, publications,periodicals, newspapers and greeting cards. It can be used by designersof product packaging and retail store environment. It can be used byinterior decorators, architects and designers. It can be used by local,state, federal, domestic and foreign governmental management of cities,states and countries. It can be used by government agencies such aspolice, military, disaster and first responder personnel and agencies.It can be used by educators, creators and publishers of text books,educational agencies and institutions. It can be used by sport teams,leagues and sport venue and event owners. It can be used performers,actors, athletes, politicians. It can be used by medical institutions,doctors, pharmaceutical and producers of other heath products. It can beused by corporations, employers and agencies. It can be used by creatorsand developers of technologies, sport equipment, real estate developersand manufacturers of vehicles and other mode of transportations.

The solution relates to a contextually intelligent communicationplatform (CICP) system and a variety of methods for its use. In oneaspect the CICP is used for acquiring information from various sourcesto facilitate a recognition of one or more objects or images usingsensor data transmitted to an augmented reality (AR) system mobileapplication software and/or an image recognition (IR) mobile applicationsoftware. Using the camera of the mobile device with the AR and or IRapplication and the device's camera, the recognized image or object ismatched with an existing data file for a known image or object residingin an image/object server in the cloud. A server in the cloud is hereindefined as a computer server located elsewhere, hosted and managed bysomeone else such as Amazon's web services. The AR system of a CICP canbe used as a method for delivering an augmentation to a user of theCICP. An Augmentation is the overlay of a digital experience such as ananimated or still 3-D model, an animation sequence, a video overlay,link to another website or information superimposed on the AR target asseen on the mobile device display screen. Several sources for an ARsystem are known. One such AR software is Vuforia™ (Qualcomm, Inc., SanDiego, Calif.). IR solutions can be used to recognize an object, matchthe image with a known image in the cloud based image server and presentan overlay of information superimposed on the image and displayed on themobile device display screen. Several commercially available IRsolutions exist including “Gimbal” offered by Qualcomm Labs, Inc., SanDiego, Calif. In one process of the present solution, a user of aconnected device receives and feeds data from one or a plurality ofsensors that share relevant information with the user and a contextuallyintelligent server in the cloud. A connected device is a hardwareincluding a Smart phone, a tablet, a computer, a laptop, AR glasses orcontact lenses, and the like. Such hardware may be connected to acommunications system such as for example, the internet, satellitecommunications, mobile telephone systems, Wi-Fi and the like. Thepresently described systems and processed, also referred to as the“solution” can process various information obtained from the sensoractivity(s) including for example, a location, a time, an activity, aspeed and direction of travel, an anxiety level assessment, a socialmedia status, a gender, a relationship status, an ambient sound or alight level, an air quality, or various other environmental aspects ofthe user. The gathered and shared information flow is an intelligenceactivity of the CICP which processes a plurality of data files relatingto a contextual environment of the user of a connected device. Theinformation acquired by the CICP about a particular user has immediateand future benefits for that user of the connected device. Thecontextual environment of a user of a connected device is defined forthe present invention as the environmental circumstances that form asetting for an event, a statement or an idea for the user and in termsin which the user's environment can be understood and assessed by theuser. This includes discovery of other devices and people in proximity,detection of any sensor system embedded in the physical world around theuser, any sensor system worn or embedded in other persons or otherdevices or equipment. It also includes the time, day and location aswell as the activity the end user is currently engaged in. For thepresent solution the contextual environment of the user of a connecteddevice is termed a context bubble. The context bubble of a user of aconnected device influences the user as a bubble of influence. As otherspeople or devices enters this contextual bubble of influence, either bythem moving into it, or the user movement so that his bubble ofinfluence encompasses the new entity(s) within the bubble, these will bedetected, acknowledged and contact established when appropriate.

In a second aspect, the CICP is used as a method for predicting a futurecontextual environmental setting for the user of the connected device.

An example of a CICP system of the present solution is depicted in FIG.1to illustrate its possible eco-architecture comprising a plurality ofdata processing stages in boxes functionally labeled and having aplurality of data transmission connections, in this example, numbered 1to 10. A user can have a connected device such as a smart phone and mayhave a combination of other additional sensors and connected devicessuch as glasses that are AR displays, headphones or other sensors thattransmit sound—audio input and a body area network (FIG. (2). A mixtureof interrelated sounds, images and sensory data experiences creates acontext bubble around the user and the context bubble becomes a bubbleof influence of the user, FIG. 2. The context bubble stage is depictedin FIG.1.

In FIG. 1, various devices within the context bubble stage send data toa context input devices-sensor input stage using connection 1. Contextinput devices—sensor input data is sent and translated to a currentcontext awareness definitions stage using connection 2. Past behaviorand actions-demographic data stage may be created using matchedinformation sent using connection 3 and connection 4. Data matchingbetween a current context awareness definitions data and a historicaland demographic data, results in data being sent from the historicaldata and demographics database using connection 4. When historical datamatches current context awareness data, then the match data is sent topredict and present definitions stage using connection 5. Predict andpresent definitions data is presented as a mobile customer relationshipmanagement (MCRM) system stage in the cloud using connection 6. In FIG.1 the MCRM system in the cloud will then bring in relevant content orinformation data from the content and experiences database stage in thecloud using connection 7. The MCRM will then present relevant content orinformation from the content and experiences database (CED) in the cloudto the user of a connected device in the context bubble using connection8. The result of presenting the relevant content or information from theCED to the user is reported back to MCRM using connection 9.

FIG. (3) depicts some of the sensors, variables and information that maybe collected and used to determine the current context in FIG. (1). FIG.(4) depicts some of the variables and information that may be collectedand used to define the current context and creating current contextawareness in FIG. (1). FIG. (5) depicts some of the variables, input andinformation that may be collected and used to determine the pastbehavior and actions in FIG. (1). FIG. (6) depicts some of the variablesand questions that may be posted, collected and used to determine thepredictions and what to present next in FIG. (1). FIG. (7) depicts someof the content, experiences and information that may be gathered, storedand used on the content and experiences database in FIG. (1).

The solution then calls for this real-time contextual information aboutthe user's present and current context to be used to determinecontextual awareness for the user. This contextual awareness includesinformation such as the user's current situation, location, activity,actions, motion, speed of travel, direction, time/day/date, who the useris—with either a fully personally identifiable tag if the user has optedin to the service—or via a not personally identifiable tag if the userhas not opted in—as well as who else is in the user's proximity and/orwhat other devices is in the user's proximity. This can be provided bytechnologies such as “Gimbal™” as provided by Qualcomm or other contextaware solutions and/or a combination of use of augmented reality andimage recognition systems see FIG. (3) for various sensor input andinformation that can be used to detect context further than describedand depicted in FIG. (10).

Some current communication solutions based on location basedtechnologies rely on GPS and other methodologies to determine a user'slocation. Others require manual “check-in” at locations to “tag” andidentify the user's location. This solution combines all of the abovewith real time contextual knowledge such as the user's activities,situations, mood, preferences and past behavior and preferences and doesnot rely on the end user actively “check in” or “tag” their location.The solution combines all the information to deliver content that iscontextually relevant to the user, real time, to the user's mobiledevice and/or other connected devices the user may have and or wear. Thesolution can use technologies such as Vuforia™ as provided and developedby Qualcomm, or other similar augmented reality solutions such asprovided by Aurasma™, Stringar™ and/or Layar™. The input, content,information and devices used in the system are depicted and defined inFIG. (3), FIG. (4), FIG. (5), FIG. (6)

The ability to monitor, verify and measure a user's activity andinteraction with a predetermined object and or image through the use ofimage recognition and/or augmented reality on the user's mobile deviceand or other connected devices the user is using or wearing provides theowner of the image and/or object information about the end user'sinteraction with the image and/or object. The solution monitors theseinteractions real time and may make the information about theinteraction such as the length of engagement, location of engagement, aswell as subsequent actions after the interaction available to the ownerof the image and or object. This is shown in FIG. (10) where a connecteddevice with a camera such as a smart phone, a tablet computer with acamera such as an iPad2™, or future devices such as augmented realityglasses and contact lenses that has image recognition software solutioninstalled. The solution architecture provides for selected images to bedistributed to visual displays such as TVs and computer monitors wherethe content can be broadcast TV, DVDs and/or videogames with sections orall of the imagery on the screen is known and stored on the targetdatabase FIG. (10). This is shown as connection FIG. (10) “1”. Thedevice will match the images in connection FIG. (10) “2” and seek amatch with known images on database FIG. (10) as shown in connectionFIG. (10) “3”. Once a match is made, the target database FIG. (10)connects with augmented reality database FIG. (10) which contains theaugmented reality experiences that will be delivered to the device anddisplayed on it as shown on connection FIG. (10) “5”. The uniqueness ofthe innovation is the connection FIG. (10) “1” with the known contentdisplayed on the visual display and how the recognition of contentdelivers new content from the AR database server to any connected devicewith a display that enables playback of augmented reality content. Ithas thereby created a connection between the broadcaster or creator ofthe video, TV or movie content, publisher, advertiser or marketer of theprinted material and/or the owner, creator, distributor, and/or retailerof the object directly to the user of the device.

It has been recognized that the ability for advertisers, marketers andpublishers to monitor interactions with their marketing andadvertisement material and images, the owner and/or manufacturer of aproduct the ability to monitor the interaction between an end user andtheir products, the ability for the creators and distributors of visualcontent such as TV broadcasts, video games, mobile applications, websites and feature films, to verify and identify who is watching andconsuming their visual content, that this ability to measure, verify,prove and replay the end user's interactions, is very valuable. Thissolution enables the advertisers, the marketers, the publishers, theproduct owners and manufactures, the creators and distributors of videogames, TV programming, movies, web sites and mobile apps the ability todo so. FIG. (10) shows how the image recognition software installed onconnected devices with cameras such as smart phones, tablet computerswith cameras and connectivity and future devices such as augmentedreality glasses and contact lenses can provide this as a use case usingthe invention's architecture. The device's camera is controlled by theimage recognition system and will look for a match on objects, printedmaterials and/or on audio visual displays of video, TV or filmcontent—connection FIG. (10) “1”. The Target database FIG. (10) containsthe predetermined targets, presents and matches these connection FIG.(10) “2” against the images detected by the device FIG. (10) “3”. TheMobile CRM system FIG. (10) is alerted when a match has beenmade—connection FIG. (10) “4”—and when the Augmented Reality experienceis superimposed on the device—connection FIG. (10) “5”. The Mobile CRMsystem FIG. (10) will then connect with the device-connection FIG. (10)“6”—and record contextual information such as the target that the useris interacting with as proven by connection FIG. (10) “1”, then logother context such as time, date, location and length of the engagementwith the recognized target. Additional contextual data as defined inFIG. (3) and FIG. (4) may also be collected by the Mobile CRM systemFIG. (10) “B”. There is wide use for the invention as a communicationplatform as shown in FIG. (17) where an ecosystem of clients, users anduse cases are depicted.

The solution also enables automatic proof of presence where theeffectiveness of marketing and advertising campaigns in print andelsewhere that contains images, visual marketing campaigns such as TVcommercials, movie trailers, call to action offers, mailed coupons,advertisements in newspapers and circular as well as coupon delivery tothe user's mobile device through all delivery mechanism, can be verifiedand proven without the need for the end user to actively “check-in” to alocation. The methodology described in [0006] can be repeated and usedfor this purpose as well. FIG. (19) shows how a location FIG. (19) cancreate a geo fence via a radial virtual fence around its location bydefining a radius FIG. (19). This geo fence will in the future bepossible to be created in other shapes than a circle around a location.Once a known user with a defined context bubble FIG. (19) enters thisgeo fence FIG. (19), a notification is sent FIG. (19)—“1” to the mobilecustomer relationship management system FIG. (19). Once this alert hasbeen matched, the mobile customer relationship management system FIG.(19) searches for the relevant information FIG. (19)—“2” on the contentdatabase FIG. (19) to present to the end user.

The solution also calls for a self-learning feature by providingfeedback that describes whether a contextually relevant offer presentedwas acted upon or not. Determining what worked and what did not, thesystem brings this information back into the past history database. Oncea similar context is detected, if the original content presented wasproven to provide the desired result, it will be presented again.Conversely, if the original content did not provide the desired result,other content and/or offers will be presented that are still relevant tothe user's context. FIG. (14) shows how the results of the content thathas been presented to the device from then mobile customer relationshipmanagement system (MCRM) FIG. (14) has a feedback loop FIG. (14) thatmeasures the actions taken on the content such as offers accepted orrejected, whether invites, offers or coupons have created the desiredresult and made the end user go to the desired location and thuscreating a proof of presence FIG. (14). The results, both positive andnegative, are reported back to the results database FIG. (14). Theresults are then reported to the current preference database FIG. (14)and then reported as historical demographic data back to the historicand preference database FIG. (14) that is feed to the mobile customerrelationship database—MCRM—FIG. (14) thus creating a complete feedbackloop and a self-learning system. FIG. (8) shows content FIG. (8) sent tothe end user's devices FIG. (8)—“1” inside the user's context bubbleFIG. (8) has a feedback loop FIG. (8)—“2” directly to the mobilecustomer relationship database FIG. (8).

The different ways to determine context may involve different sensorinput on a wide diversity of technical capabilities on mobile devices,on peer-to-peer enabled devices, on embedded sensors in accessories,clothing, glasses, microphones, headsets, gloves, shoes, watches,tablets as well as embedded sensors in the surrounding environment,devices in proximity and/or in transportation devices. Some may alsoinclude the ability to detect hostile environments such as pollutantsand or toxins, unhealthy lighting or sound levels, elevated stress andanxiety levels. Other use cases could thereby be envisioned for thesolution. As an example, the security capabilities may be of value tolaw enforcement, homeland defense, military and/or security forces inmonitoring personnel or civilians as they are entering hostileenvironments, or rescue workers entering dangerous areas hit by naturaland other disasters. The contextual information is gathered from thevarious devices inside the end user's context bubble as shown in FIG.(1) connection “1”. The contextual information is then transmitted FIG.(1)—“2” to create current context awareness FIG. (1). FIG. (3) lists avariety of sensor input and information that will be presented to thecontext input devices FIG. (3) to create current context awareness. Thesolution may use some, all or a variety of these sensor input, and willinclude future sensor input as they become available or developed.

Different scenarios for communication between the user's devices and thecloud based databases as well as the MCRM are envisioned. CurrentWireless Local Area Network (WLAN) as well as LAN such as Wi-Fi ispossible. In addition, the ability for the user's devices to connectwith other devices in proximity by establishing ad-hoc peer-to-peernetworks, Near Field Communication (NFC) protocols as well as othermethods such as Bluetooth™ is included. This enables the solution to beadaptable to any future protocols for communications between devices andthe cloud based servers and MCRM as well as other sensors and devices inproximity. Multiple commercially available protocols exist for theestablishment of the peer to peer connection and subsequentcommunication such as BlueTooth™ and Qualcomm's AllJoyn™ technology. Ascenario for peer-to-peer communication is depicted in FIG. (9). An enduser's context bubble FIG. (9) has been created and defined as a bubbleof influence, FIG. (9). Another user or multiple users—other users FIG.(9) enter the bubble of influence with their context bubble—FIG. (9)context bubble. Once inside the bubble of influence of the first enduser FIG. (9), a communication link is established between the firstuser and the other users, peer to peer FIG. (9). Conversely, the processis repeated for any other connected device, other devices FIG. (9) thatenters the bubble of influence of the first end user, FIG. (9). A peerto peer communication link is then established between the devices, FIG.(9).

The solution calls for the delivery of content to the user's device tobe across multiple methods. The information can be simple push SMSmessages, web links, coupon images and simple call to action offers. Theinformation can also be delivered inside video games, AR experiences orother audio visual representations on the user's devices. Theinformation can also be used to control any of the above such as thereal time activity of sports athletes and or sporting equipment such asrace cars and boats, and/or the real-life ball trajectory of a ball in asporting game in a stadium. In such a case, real-time telemetry, balland player tracking, motion traction and other sensor input can be usedat the place of game play, on equipment or devices such as cars andboats. In addition, player conditions such as heart rates, hydrationlevels and more can be collected via embedded sensors, transmitted to acloud based server, monitored and collected. The data can then be usedto control in-game 3D models depicting the actual game play, boat/carmotion and telemetry, ball travel and player positions. FIG. (20)depicts such a system. FIG. (20) shows how the image recognitionsoftware installed on connected devices with cameras such as smartphones, tablet computers with cameras and connectivity and futuredevices such as augmented reality glasses and contact lenses can providethis as a use case using the invention's architecture. The device'scamera is controlled by the image recognition system and will look for amatch on objects, printed materials and/or on audio visual displays ofvideo, TV or film content—connection FIG. (20) “1”. The target databaseFIG. (20) contains the predetermined targets, presents and matches theseconnection FIG. (20) “2” against the images detected by the device FIG.(20) “3”. The Mobile CRM system FIG. (20) is alerted when a match hasbeen made—connection FIG. (20) “4”—and when the Augmented Realityexperience is superimposed on the device—connection FIG. (20) “5” withreal-time, real life telemetry information collected at the sport eventvia the sport telemetry information FIG. (20) and presented to theaugmented reality experiences database FIG. (20) in connection “7”. Thedata presented from the sporting event telemetry database can be used tocontrol avatars of players and equipment in augmented reality renderingand/or sporting game associated with the sporting event, based on reallife actions, movements, speed and locations collected from the reallife event. The Mobile CRM system FIG. (20) connects with the device-connection FIG. (20) “6”—and records contextual information such as thetarget that the user is interacting with as proven by connection FIG.(20) “1”, then log other context such as time, date, location and lengthof the engagement with the recognized target. Additional contextual dataas defined in FIG. (3) and FIG. (4) may also be collected by the MobileCRM system FIG. (20).

The solution calls for the content to be displayed when the user is inthe right context, not only location. Hence, information about aspecific sporting game with information such as score, player and balllocations, trajectories, activities, player conditions can be deliveredto fans that are in the context of the stadium, as well as users thatare watching the game on TV away from the stadium as well as fansreading a publication covering the sport anywhere using their connecteddevice. A methodology to connect the fans away from the actual sportingevent is described in [0012].

The ability to collect sensor input from a real live event may in thefuture be able to be used in tactile feedback on the users sensorenabled clothing, seating and or environment. One such use case may becreated around a sporting event. As shown in in FIG. (21) data collectedfrom a sporting event can be collected and sent to a sporting eventtelemetry database. The data presented in connection “8” may containsuch tactile information as gathered by sensors worn by players and sentas part of the sporting event telemetry information FIG. (21). Thisinformation may then be combined with the other information presented bythe augmented reality experience database FIG. (21) and presented to thedelivered in FIG. (21) to the connected device, FIG. (21) which may beconnected with tactile feedback sensors and/or connected via apeer-to-peer communication as shown as connection FIG. (21) “9” tosensors worn by the end user that provides the tactile feedback, tactilefeedback sensors and audio/video displays worn by the end user, FIG.(21). [0014] The ability to collect visual information from a real lifeevent, combining this with motion tracking and other sensor input willenable a real life reproduction of the event to be presented to the enduser when relevant using technologies such as augmented reality, head-updisplays, augmented reality enabled glasses and contact lenses as wellas permanent installations. FIG. (20) depicts such a system. FIG. (20)shows how the image recognition software installed on connected deviceswith cameras such as smart phones, tablet computers with cameras andconnectivity and future devices such as augmented reality glasses andcontact lenses can provide this as a use case using the invention'sarchitecture. The device's camera is controlled by the image recognitionsystem and will look for a match on objects, printed materials and/or onaudio visual displays of video, TV or film content—connection FIG. (20)“1”. The Target database FIG. (20) contains the predetermined targets,presents and matches these connection FIG. (20) “2” against the imagesdetected by the device FIG. (20) “3”. The Mobile CRM system FIG. (20) isalerted when a match has been made—connection FIG. (20) “4”—and when theAugmented Reality experience is superimposed on the device—connectionFIG. (20) “5” with real-time, real life telemetry information collectedat the sport event via the sport telemetry information FIG. (20) andpresented to the augmented reality experiences database FIG. (20) inconnection “7”. The data presented from the sporting event telemetrydatabase can be used to control avatars of players and equipment inaugmented reality rendering and/or sporting game associated with thesporting event, based on real life actions, movements, speed andlocations collected from the real life event. The Mobile CRM system FIG.(20) connects with the device-connection FIG. (20) “6”—and recordcontextual information such as the target that the user is interactingwith as proven by connection FIG. (20) “1”, then log other context suchas time, date, location and length of the engagement with the recognizedtarget. Additional contextual data as defined in FIG. (3) and FIG. (4)may also be collected by the Mobile CRM system FIG. (20).

The solution enables the experiences that are presented to becontrolled, the results measured and proven, thus creating an ability tocommercialize the communication channels and the activities that are aresult of the communication. The client eco system depicted in FIG. (17)shows the wide potential user base of this invention and/or portions ofit once it is commercialized and widely available.

The solution creates contextual wisdom be presenting relevant contentsuch as information, entertainment, warnings and/or offers to the rightperson while they are in the right context. This process is shown inFIG. (15) and FIG. (16). FIG. (15) depicts how contextual information isprovided and detected and presented to the MCRM—mobile customerrelationship management system FIG. (15) then combined with discoveryand historic data. Proximity information is also presented into the MCRMsystem—FIG. (15) through sensors. Current and future information such asinference and prediction, personalized offer and content is also shownto be connected to the MCRM—FIG. (15). The process of gettinginformation, gather it, match it, refine it and present it is shown tocreate wisdom by presenting the right information to the right end user,FIG. (15). By providing this information, a contextually intelligentcommunication is enabled, FIG. (15). The process to collect data, createmeaning of information, creating knowledge from the information,combining it with current and past information, understanding value,creating wisdom to present the right information to the right person atthe right time and place is shown in FIG. (16).

FIG. (1) depicts an overview of the eco-system architecture. Variousdevices within the context bubble FIG. (1) gathers information as shownin FIG. (1) connection “1”. The information is translated to currentcontext awareness definitions, FIG. (1), connection “2”. Past behaviorand actions FIG. (1) are determined by combining the current contextawareness definitions as transmitted by FIG. (1)—“3” and matched withhistorical and demographic data as stored in the historical data anddemographics database, FIG. (1) and transmitted if a match is made inFIG. (1) “4”. Once the historical data FIG. (1)-4 and past behavior FIG.(1) is combined with the current context awareness FIG. (1), decisionson what to predict and present, FIG. (1) can be presented FIG. (1)—6 toa mobile customer relationship management system in the cloud FIG. (1).This mobile customer relationship management system in the cloud FIG.(1) will then bring in relevant content and/or information from thecontent and experiences database in the cloud FIG. (1) in connectionFIG. (1)—“7”. The mobile customer relationship management system in thecloud FIG. (1) will then present relevant content and/or informationfrom the content and experiences database in the cloud FIG. (1) inconnection FIG. (1) to the end user in the context bubble FIG. (1) “8”.The results of the communication presented in communication “8” FIG. (1)is reported back to the mobile customer relationship management serverFIG. (1).

FIG. 2 shows the contextual bubble where a user is surrounded by avirtual bubble the is created around the user from the user's variousconnected devices such as mobile devices, augmented reality visualdisplays such as AR glasses, audio display devices, body area sensorsworn in clothing, accessories and shoes. It also shows that others canbe in the same context bubble such as other users or other connecteddevices. Once these are in the same context bubble, they become part ofthe bubble of influence and communication between the first user and theother users and or devices can be established via peer-to-peer or anyother near field communication. As show in FIG. (2) an end user has aconnected device FIG. (2) such as a smart phone and may have acombination of other additional sensors and connected devices such asglasses that are AR displays, FIG. (2), headphones or other sensors thattransmit sound—audio input FIG. (2) and a body area network FIG. (2).This creates a context bubble FIG. (2) around the end user and thebubble becomes a bubble of influence, FIG. (2).

FIG. (3) depicts some of the sensors, variables and information that maybe collected and used to determine the current context in FIG. (1).

FIG. (4) depicts some of the variables and information that may becollected and used to define the current context and creating currentcontext awareness in FIG. (1)

FIG. (5) depicts some of the variables, input and information that maybe collected and used to determine the past behavior and actions in FIG.(1).

FIG. (6) depicts some of the variables and questions that may be posted,collected and used to determine the predictions and what to present nextin FIG. (1).

FIG. (7) depicts some of the content, experiences and information thatmay be gathered, stored and used on the content and experiences databasein FIG. (1)

FIG. (8) shows content FIG. (8) sent to the end user's devices FIG.(8)—“1” inside the user's context bubble FIG. (8) has a feedback loopFIG. (8)—“2” directly to the mobile customer relationship database FIG.(8).

FIG. (9). A scenario for peer-to-peer communication is depicted in FIG.(9). An end user's context bubble FIG. (9) has been created and definedas a bubble of influence, FIG. (9). Another user or multiple users—otherusers FIG. (9) enter the bubble of influence with their contextbubble—FIG. (9) context bubble. Once inside the bubble of influence ofthe first end user FIG. (9), a communication link is established betweenthe first user and the other users, peer to peer FIG. (9). Conversely,the process is repeated for any other connected device, other devicesFIG. (9) that enters the bubble of influence of the first end user, FIG.(9). A peer to peer communication link is then established between thedevices, FIG. (9).

FIG. (10) shows how the image recognition software installed onconnected devices with cameras such as smart phones, tablet computerswith cameras and connectivity and future devices such as augmentedreality glasses and contact lenses can provide this as a use case usingthe inventions architecture. The device's camera is controlled by theimage recognition system and will look for a match on objects, printedmaterials and/or on audio visual displays of video, TV or filmcontent—connection FIG. (10) “1”. The Target database FIG. (10) containsthe predetermined targets, presents and matches these connection FIG.(10) “2” against the images detected by the device FIG. (10) “3”. TheMobile CRM system FIG. (10) is alerted when a match has beenmade—connection FIG. (10) “4”—and when the Augmented Reality experienceis superimposed on the device—connection FIG. (10) “5”. The Mobile CRMsystem FIG. (10) will then connect with the device-connection FIG. (10)“6”—and record contextual information such as the target that the useris interacting with as proven by connection FIG. (10) “1”, then logother context such as time, date, location and length of the engagementwith the recognized target.

FIG. (11) shows a high level overview of the architecture of informationfrom context awareness to content delivery and the interaction with astandard commercially available big data management solution such asHoodad™ and others.

FIG. (12) show more details of the various databases and sensor inputpoints between the sensors in the context bubble that determines theuser's context and how the various databases feed information into theMCRM system to ensure delivery of the relevant information

FIG. (13) show an overview of others and other devices that aredetermined to be in proximity to the user inside the user's contextbubble and then how a connection between these users and or devices canbe tied into the MCRM system so that relevant information and contentcan be delivered to these as well if of relevance

FIG. (14) shows how the results of the content that has been presentedto the device from then mobile customer relationship management system(MCRM) FIG. (14) has a feedback loop FIG. (14) that measures the actionstaken on the content such as offers accepted or rejected, whetherinvites, offers or coupons have created the desired result and made theend user go to the desired location and thus creating a proof ofpresence FIG. (14). The results, both positive and negative, arereported back to the results database FIG. (14). The results are thenreported to the current preference database FIG. (14) and then reportedas historical demographic data back to the historic and preferencedatabase FIG. (14) that is feed to the mobile customer relationshipdatabase—MCRM—FIG. (14) thus creating a complete feedback loop and aself-learning system.

FIG. (15) shows a schematic overview of the information flow fromcontext devices such as sensors and infrastructures, past and historicaldata, combined with context that is detected and discovered withhistorical data, inference and predicted desired results andpersonalized offers are flowing in the path of information gatheringfirst, then matching and refining the data, then present it to thedevice thus creating contextual wisdom. The path of information andcontent flow is to first define and collect information from the varietyof input devices both on the mobile device but also in its proximity andcontext bubble, then gather this information, match the real timecontextual data with historical and demographical data as well as pastbehavior, historical data and preferences both inferred and recorded,then combine this to refine the results of the content and informationto be presented to the mobile device, this creating wisdom of thecontent delivery.

FIG. (16) depicts how contextual information is provided and detectedand presented to the MCRM—mobile customer relationship management systemFIG. (16) then combined with discovery and historic data. Proximityinformation is also presented into the MCRM system—FIG. (16) throughsensors. Current and future information such as inference andprediction, personalized offer and content is also shown to be connectedto the MCRM—FIG. (16). The process of getting information, gather it,match it, refine it and present it is shown to create wisdom bypresenting the right information to the right end user, FIG. (16). Byproviding this information, a contextually intelligent communication isenabled, FIG. (16).

FIG. (17) show an outline of the methodology of evaluations used toperform progressive steps in determining best content to presentcreating contextual wisdom. The process to collect data, create meaningof information, creating knowledge from the information, combining itwith current and past information, understanding value, creating wisdomto present the right information to the right person at the right timeand place is shown in FIG. (17).

FIG. (18) depicts and shows that there is wide use for the solution as acommunication platform as shown in FIG. (18) where an ecosystem ofclients, users and use cases are depicted.

FIG. (19) show an overview of the methodology to a method fordetermining the context a person is currently in by using object andimage recognition system where the augmentation is delivered once theimage and/or object has been recognized and matched against a knownimage and/or object in an image/object server in the cloud. In oneembodiment of the solution the method of creating a contextualintelligent communication platform involves image recognition andaugmented reality technology commercially available software known toone of skill in the art such as Vuforia™ version 1.0 and more advanceforms such as version 1.5 and future versions (Qualcomm, San Diego,Calif.), or similar solutions provided by augmented reality companiessuch as Aurasma, Blippar, Layar, Stringar and or others.

FIG. (20) shows how a location FIG. (20) can create a geo fence via aradial virtual fence around its location by defining a radius FIG. (20).Once a known user with a defined context bubble FIG. (20) enters thisgeo fence FIG. (20), a notification is sent FIG. (20)—“1” to the mobilecustomer relationship management system FIG. (20). Once this alert hasbeen matched, the mobile customer relationship management system FIG.(20) searched for the relevant information FIG. (20)—“2” on the contentdatabase FIG. (20) to present to the end user. The virtual fence isdepicted as a circle by may also be created in other forms and shapes tobetter adapt to the physical landscape and buildings around thelocation.

FIG. (21) shows how the image recognition software installed onconnected devices with cameras such as smart phones, tablet computerswith cameras and connectivity and future devices such as augmentedreality glasses and contact lenses can provide this as a use case usingthe inventions architecture. The device's camera is controlled by theimage recognition system and will look for a match on objects, printedmaterials and/or on audio visual displays of video, TV or filmcontent—connection FIG. (21) “1”. The Target database FIG. (21) containsthe predetermined targets, presents and matches these connection FIG.(21) “2” against the images detected by the device FIG. (21) “3”. TheMobile CRM system FIG. (21) is alerted when a match has beenmade—connection FIG. (21) “4”—and when the Augmented Reality experienceis superimposed on the device—connection FIG. (21) “5” with real-time,real life telemetry information collected at the sport event via thesport telemetry information FIG. (21) and presented to the augmentedreality experiences database FIG. (21) in connection “7”. The datapresented from the sporting event telemetry database can be used tocontrol avatars of players and equipment in augmented reality renderingand/or sporting game associated with the sporting event, based on reallife actions, movements, speed and locations collected from the reallife event. The Mobile CRM system FIG. (21) connects with the device-connection FIG. (21) “6”—and record contextual information such as thetarget that the user is interacting with as proven by connection FIG.(21) “1”, then log other context such as time, date, location and lengthof the engagement with the recognized target. Additional contextual dataas defined in FIG. (3) and FIG. (4) may also be collected by the MobileCRM system FIG. (21).

FIG. (22) shows how data collected from a sporting event can becollected and sent to a sporting event telemetry database. The datapresented in connection “8” may contain such tactile information asgathered by sensors worn by players and sent as part of the sportingevent telemetry information FIG. (22). This information may then becombined with the other information presented by the augmented realityexperience database FIG. (22) and presented to the delivered in FIG.(22) to the connected device, FIG. (22) which may be connected withtactile feedback sensors and/or connected via a peer-to-peercommunication as shown as connection FIG. (22) “9”to sensors worn by theend user that provides the tactile feedback, tactile feedback sensorsand audio/video displays worn by the end user, FIG. (22).

FIG. (23) In one embodiment of the solution, a complete end-to-endcommunication solution is envisioned for a stadium owner and theattendees that are attending events in the stadium. FIG. (23) “1” showshow AR and or IR can be used to engage and capture the information abouta person who has bought a ticket to the event and plan to attend itwhile at home or before the event. This is done by creating an AR or IRexperience that is triggered by the specific tickets for entrance to thespecific event. This can be an AR game such as a soccer penalty kickgame, basketball free throw game, or a baseball bating versus pitchergame, basically anything that is related to the sport being played inthe stadium that the ticket will provide entrance to. When the user isstarting such an application on their mobile device that contains theimage of this ticket as part of its image recognition database thatserves as the base for the AR experience that is superimposed upon it,it is recognized that the user is interacting with that specific ticket,where the user is located at that given time, and the identification ofthe device is established. This information is captured and transmittedto the cloud based MCRM system. Once the system recognizes that the useris coming to the event, additional information about the event,reminders, offers, upsell offers such as bring a friend and get adiscount, share on Facebook or other social network that you are goingand earn rewards. FIG. [23] “2” shows that as the user is in transit onthe way to the event. This is done by locating the user via the user'smobile device and matching it with the time and date of the event whichis known because of the capture in FIG. [23] “1”. As the user isapproaching the stadium, offers, coupons, information and encouragementscan be delivered. Offers can be for discounts at concessions inside thevenue, merchandise, limited time offers, memorabilia and/or items suchas clapper or signs that the stadium owner wants their fans to displayduring the event. Offers can also be delivered on behalf ofparticipating sponsors outside the stadium, such as bars andrestaurants. These can be timed so that they are only valid for acertain period before the event, and may offer other limitations such asthe requirement to display the ticket to the event in order to redeemthe offer. Coupons can be delivered in the same fashion—both formerchandise inside the stadiums as well as outside pre-event.Information can also be delivered as the user approaches the stadium.This can be dynamic updates such as where there is open parking, whatentry gate to the stadium has the shortest line, and updates on thestatus of the home team playing. Encouragements can also be delivered tospur fan excitement such as be loud, chant or other enticement to cheerthe home team on. FIG. [23] “3” shows how more specific hyper-locationoffers can be presented. “Hyper-location” is here defined as theknowledge of a location in a very specific location such as the specificstadium and places within it, and around the stadium. For instance, thesystem is able to pin-point where the car was parked. This can be donemanually, or used by determining rate of speed and time stationary whichcan trigger an active response such as a question—“have you parked yourcar?” Based on hyper location such as which gate the user entered thestadium through, offers that are based on concessions stands that areclose by that specific gate can be served as the user enters the gate.Using AR triggers as described elsewhere as directional indicators,directional information can be offered to the user. Excusive content andAR experiences can be presented and used to drive the user to specificlocations within the stadium such as the team store, museums or otherlocations where the stadium owner wants to present enhanced experiences.These can be time dependent and only be active before and after the gameplay—and/or during breaks. FIG. [22] “4” shows how a game experienceand/or AR experience can be enabled that is controlled and limited bytime and location. One such embodiment is the ability to take an ARenhanced photo of the user where an avatar image or real life image ofone of the home team's star players is superimposed on the photo next tothe user making the illusion that the user was standing next to thestar. Rewards to the user for sharing this on the user's social networkscan be presented. Other game experiences can be triggered as well. Thesecan be location based; proximity based and/or can be games played amongand between users. Technologies such as AllJoyn peer-to-peer solutionscan be used for this. One aspect could be a game that is played betweenfans in the stands which is only active during breaks—such as virtualbeach ball tossing between fans—those who ends up with it when the musicends wins, and they are displayed on the big score board. Other gamescould be created to get fans to rally when the team needs support, othergames could create images on the phones that when the phones are held upwill form a mosaic of a message/image the stadium owner wants todisplay. Other games could be crowdsourcing photos into a mosaic that isdisplayed on the scoreboard—and transmitted to those watching the gameat home and via the fans social networks. Other embodiments of thesolution is where the knowledge that a user is in the stadium, and thatthe user is approaching a concession stand that has a context beacon asdescribed before, the concession stand's menu can be transmitted andpresented on the user's mobile device as the user approaches. The usercan then order from the mobile device, the order will be transmitted tothe concession stand's POS system and the order filled. When the usergoes to an established and dedicated mobile order line, the order isalready ready for the user to pick up. The user will be able to use amobile payment solution to pay for the order without having to presentany other form of payment and the concessions stand staff is free toattend to other patrons. In the future, it is envisioned that the devicecan capture activity levels and noise levels. The stadium owner canthereby measure who is active, and who is loud cheering for the hometime, and at what time they are doing so. Those that spur the home teamon with chants and active participation can thereby be rewarded fortheir involvement with the live action game based on their physicalactivity and noise level. In other instances, as the user moves along inthe stadium, contests can be presented and offered at the right time andlocations with relevant prizes and offers to the end user. One suchenvisioned game can be a “scavenger hunt” or “treasure hunt” where theuser will be presented quizzes and puzzles that are relevant to thestadium and the team playing and the prizes will provide value to theend user based on their location and activity. These games can only beplayed while inside the stadium and only when there is a sporting eventor other event. As the user travels through the environment and solvesthe puzzles, riddles and quizzes, new clues will be revealed based onlocation, whether an image has been found and IR or AR is used torecognize that the user is at that specific location, another clue willbe unlocked and the user is presented the opportunity to solve the nextquiz, riddle or puzzle. This repeats until the user ends at the finallocation or has found the right item that triggers the unlocking of thefinal prize. The final prize can be a ticket to a future event,back-stage passes, VIP treatment, meet players, receive coupons,exclusive content, physical or digital merchandise, games, video oraudio content. As the user continues through the physical environment,automatic updates to the user's context and surroundings can be sharedwith others based on the user's permissions. These can be updates totheir social media status such as on Facebook and Twitter for anindividual end-user. Should there be other end-users in the proximity toan end-user and all have at one time downloaded one of the applicationsthat use the invention's methodology, an ad-hoc peer-to-peercommunication can be established using technologies such as Qualcomm' s“AllJoyn” solution. Another embodiment of the solution is the use ofthis technology for visitors to stadiums where there is a long linewaiting to enter or line up for special events, greet the players orlocker-room access. In such a context, a location based game can betriggered and end users that have the solution on their devices can beconnected and start playing against each other. The winner may get afront of the line pass if he wins. As people move away from thelocation, they will drop out of the game, and others will be invited tojoin as they come into proximity. The game can only be played by usersthat are in the specific location, which is the context defined by theback-end MCRM system. Another use is for the user who is in the stadiumand has had the application downloaded and is approaching a concessionstand, the ad hoc proximity peer-to-peer system will create a connectionwith the concessions stand's mobile device/and or connected point ofsale (POS) system. The menu from the concessions stand will be displayedon the user's mobile device. The user can select the menu items and theorder will be transmitted to the concessions stand's mobile deviceand/or connected POS system. Once the order has been placed, a cloudbased mobile payment system such as Qualcomm, Inc.'s “PAY” solution ornear field communication (NFC) payment solution such as Google “Wallet”can be used to complete the transaction. When “PAY” is used, the user'spurchase is captured and retained in the MCRM system. In other instancesas a person is moving around their surroundings, images can becomeroadmaps and markers. These can offer directional feedback to theend-user. They can be simple logos, arrows, or any images that the IR/ARtechnology recognizes as a directional sign as matched against the cloudbased target database. The image can indicate to one end user to go leftbased on that person's desired destination, the same image can indicateto another user to go right based on the person's desired destinationthat differs from the first person's destination. Content that iscaptured in the stadium by various cameras can be transmitted directlyto the devices in the stadium. This can be imagery such as alternativecamera angles, re-play, re-wind, or alternative point of view fromembedded cameras on referee's caps or players helmets. Once sensors areplaced and embedded in player's gear such as shoes, helmets, protectivegear or equipment, information from these sensors can also betransmitted to the user's device while inside the stadium through WiFi,WLAN, LAN and or Distributed Antenna Systems (DAS). Since broadcastingonly inside the stadium to fans in the stands, this should not interfereor violate broader broadcasting rights held by others. Anotherembodiment of the application is to use the AR targets as communicationchannels. Images placed on items such as, but not limited to, apparel,photos, ads, logos, products, packaging, posters will become adistribution channel of digital content that can be superimposed ontothe target. This information can be content such as news, updates,announcements, entertainment, encouragement, supportive messages,reminders and alerts that are relevant to the product that has the imageon it or is depicted in the image or is associated with the image.Personalization of the content can be created by the MCRM back-endsystem that drives and delivers the messaging in the AR experience fromthe AR database, combined with the user's current and past contexts andpast behavior, preferences and behavior. Traditional maps for stadiumscan also serve as dynamic AR targets. A use of the solution is theability to create interactive, dynamic and personalized AR experiencessuperimposed on the printed map. This can feature information, offers,coupons and details about locations and concessions stands, it canprovide directions to locations and present high lights on the routechosen. The AR map can then be ported onto the device and remain on thedevice with the user's location updated as the user proceeds along theselected route. The solution can also be used to create automaticcheck-in and registration for users that enters a specific locationwhere a payment, registration or notification of the user's presence isrequired. One such use may be for attendees at the stadium. Here theuser would have registered prior to arriving by purchasing the entryticket, experienced an AR enhancement associated with the ticket asmentioned before. The MCRM system registers this booking, and when theuser crosses the geo-fence established by the location, the MCRMregisters this and sends a message to the order, booking, andregistration and/or ticket office system at the physical location. Incertain physical locations where GPS signals are weak—such as indoors,event centers, casinos and in large shopping malls—the solution enablesAR/IR targets to become virtual guides and sign posts. By recognizingtargets that are placed in a known physical location, the MCRM systemwill capture the target and match it against the target database. Sincethe physical location of the target is known, the user's location isalso known since the user needs to be in front of the target for theAR/IR to recognize it. The MCRM system can then send information aboutother known physical places in proximity to the current AR/IR target.Information such as directions, landmarks, distance, features of thelocation and offers from the location can be displayed. These will besuperimposed on a digital map unlike the traditional AR experienceswhere you hold your phone up against the horizon. These systems aretypically dependent on GPS coordinates of the user and the AR overlaysand do not work indoors. The solution uses a combination of the user'sknown location, the MCRM database of known and defined locations, theMCRM's knowledge of AR/IR target's locations, and if a known number ofthese AR/IR targets are recognized as dynamic directional markers,directional arrows can appear as the user moves through the environmentguiding the user to the desired end location. The solution also enablesdigital signs to display relevant information as a user moves throughthe stadium. The display sign is connected with the MCRM system in thecloud. Once a user's context bubble enters the “sphere of influence” ofthe digital sign, the user's mobile device will recognize the event. TheMCRM system will recognize the event from both the digital display aswell as the user. The MCRM will send an appropriate message to bedisplayed on the display based on the users past preferences, currentstate of mind, the user's likely next destination as well as anyrecorded activities in the future. A user's current context can also bedefined by the user's location. There are numerous means to define andrecord this, and the solution is not dependent upon any specific method.As the user is moving about in the physical world, the user's contextwill constantly change. The user can use the MCRM to specifically recordcertain aspects of the context and share it with others. One suchembodiment is to place a tag on the specific location a user is in, andthen share this location with others either directly to one otherperson, or to that person through social media, or share with a group ofothers. Another embodiment of the solution is the use for liveperformances such as concerts, circus acts and theater performanceswithin the stadium. Aspects within the performance, game or event suchas props, stands, displays, mannequins, statues, dolls can serve as ARmarkers and pre-recorded acts can be superimposed on them before, duringand after the sporting event. This can be AR overlay that is common andseen the same way by all in the audience, or it can be customized to berandomly displayed or specifically presented based on the end user'sdemographics such as age and or gender. The stadium sporting event canalso be enhanced with AR superimposed content that can be same for allthe users in the audience, randomly presented or customized for specificaudience members. Certain locations can be enhanced with contextuallyspecific features. One such example is an AR experience that enables andenhances betting and wagering within horse race tracks, stadiums and/orcasinos. Offers that are specific to such locations can be automaticallypresented to users that are in such a context and have a desire toreceive them. Within other venues, behavior that is beneficial to thevenue can be rewarded. Loyalty programs, special front of the lineentrance and other VIP treatment can be awarded to users that are in thecontext and promote to their social network that they are there. Shouldtheir friends respond in a way that is beneficial to the venue owner,such user will receive the benefits, status and loyalty rewards FIG.[23] “5” shows that as the user is leaving the stadium, rewards forspecific behavior can be presented. The action of leaving can bedetermined by crossing the geo-fence as well as time. Activities such assharing the experience with others, participating in games, chanting andspurring the home team on can be rewarded. The rewards can be for offersto get a product such as team merchandise on the way out, or get abottle of water for the ride home. Information can also be presented atthis time such as traffic updates, where the car was parked, crowds onpublic transportations, altercations and traffic accidents. FIG. [23]“6” shows when the user is in transit on the way away from the stadium,information, content and offers can be presented. They can be a “thankyou for coming” note, can be exclusive high light videos and contentfrom the game or offers to go to participating restaurants, bars orretailers on the way home. These can all be sponsored by the team, thestadium owner or their sponsors. They can be for products, discountoffers or merchandise from the team. FIG. [23] “7” shows that when theuser is back at the location as detected in FIG. [23] “1” or anotherlocation that has been determined to be the user's home, additionaloffers, coupons, information and content can be presented by the teamand/or stadium owners. This can updates on the next time the team isplaying in the stadium, discount offers such as bring a friend and get50% off or other enticement offers to return to the stadium. Thiscomplete solution thereby enables the stadium owner and/or team tocommunicate with their fans from the moment they purchase their tickets,as they are coming to the stadium, while they are attending the sportingevent at the stadium, as they are returning on the way home and afterthey have returned home, thus creating loyalty between the fan and thestadium/team.0042

FIG. [24] In one embodiment of the solution, a complete end-to-endcommunication solution is envisioned for a theme park owner/operator andthe attendees that are attending the park. FIG. [24] “1” shows how ARand or IR can be used to engage and capture the information about aperson who has bought a ticket to the park and plan to attend it whileat home or before the event. This is done by creating an AR or IRexperience that is triggered by the specific tickets for entrance to thespecific theme park. This can be an AR game that is related to the themepark that the ticket will provide entrance to. When the user is startingsuch an application on their mobile device that contains the image ofthis ticket as part of its image recognition database that serves as thebase for the AR experience that is superimposed upon it, it isrecognized that the user is interacting with that specific ticket, wherethe user is located at that given time, and the identification of thedevice is established. Once the system recognizes that the user iscoming to the park, additional information about the event, reminders,offers, upsell offers such as bring a friend and get a discount, shareon Facebook or other social network informing others that the user isgoing and the user will earn rewards for this. FIG. [23] “2” shows thatas the user is in transit on the way to the park. This is done bylocating the user via the user's mobile device and matching it with thetime and date of the ticket which is know because of the capture in FIG.[24] “1”. As the user is approaching the park, offers, coupons,information and encouragements can be delivered. Offers can be fordiscounts at concessions inside the park, restaurants, merchandise,limited time offers, memorabilia and/or items such as dolls, mascots,hats and masks. Offers can also be delivered on behalf of participatingsponsors outside the park, such as hotels, bars and restaurants. Thesecan be timed so that they are only valid for a certain period before thepark opens, and may offer other limitations such as the requirement todisplay the ticket to the park to redeem the offer. Coupons can bedelivered in the same fashion—both for merchandise inside the park aswell as outside pre-entry. Information can also be delivered as the userapproaches the park. This can be dynamic updates such as where there isopen parking, what gate has the shortest line, and updates on the statusof length of lines at different lines. FIG. [24] “3” shows how morespecific hyper-location offers can be presented. “Hyper-location” ishere defined as the knowledge of a location in a very specific locationsuch as the specific park and places within it. The system can pin-pointwhere the car was parked. This can be done manually, or used bydetermining rate of speed and time stationary which can trigger anactive response such as a question—“have you parked your car?” Based onhyper location such as which gate the user entered the park through,offers that are based on concessions stands that are close by thatspecific gate can be served as the user enters the gate. Using ARtriggers as described elsewhere as directional indicators; directionscan be offered to the user while inside the park. Excusive content andAR experiences can be presented and used to drive the user to specificlocations within the park such as the specific rides, restaurants,concession stands or other locations where the park owner wants topresent enhanced experiences. These can be time dependent and only beactive before and after the game play—and/or during breaks. FIG. [24]“4” shows how a game experience and/or AR experience can be enabled thatis controlled and limited by time and location. One such embodiment isthe ability to take an AR enhance photo of the user where an avatarimage of one of the parks mascots or action figure is superimposed onthe photo next to the user making the illusion that the user wasstanding next to the mascot or action figure. Rewards for sharing thison the user's social networks can be presented. Other game experiencescan be triggered as well. These can be location based and/or proximitybased and can be games between users. Other games could be crowdsourcingphotos into a mosaic that is displayed on a digital scoreboard. Otherembodiments of the invention is where the knowledge that a user is inthe park, and that the user is approaching a concession stand that has acontext beacon as described before, the concession stand's menu can betransmitted and presented on the user's mobile device as the userapproaches. The user can order from the mobile device, the order will betransmitted to the concession stand's POS system and the order filled.When the user goes to the dedicated mobile order line, the order isalready ready for the user to pick up. The user will be able to use amobile payment solution to pay for the order without having to presentany other form of payment and the concessions stand staff is free toattend to other patrons. As the user moves along in the park, contestscan be presented and offered at the right time and location withrelevant prizes and offers to the end user. One such game can be a“scavenger hunt” or “treasure hunt” where the user will be presentedquizzes and puzzles that are relevant to the park and the prizes willprovide value to the end user based on their location and activity. Asthe user travels through the environment and solves the puzzles, riddlesand quizzes, new clues will be revealed based on location, whether animage has been found and IR or AR is used to recognize that the user isat that specific location, another clue will be unlocked and the user ispresented the opportunity to solve the next quiz, riddle or puzzle. Thisrepeats until the user ends at the final location or has found the rightitem that triggers the unlocking of the final prize. The final prize canbe a ticket to a future visit, front of line passes, VIP treatment,receive coupons, exclusive content, physical or digital merchandise,games, video or audio content. As the user continues through thephysical environment, automatic updates to the user's context andsurroundings can be shared with others based on the user's permissions.These can be updates to their social media status such as on Facebookand Twitter for an individual end-user. Should there be other end-usersin the proximity to an end-user and all have at one time downloaded oneof the applications that use the solution's methodology, an ad-hocpeer-to-peer communication can be established using technologies such asQualcomm's “AllJoyn” solution. One embodiment of the solution is the useof this technology for visitors inside the park where there are longlines waiting to enter rides or special events. In such a context, alocation based game can be triggered and end users that have thesolution on their devices can be connected and start playing againsteach other. The winner may get a front of the line pass if he wins. Aspeople move away from the location, they will drop out of the game, andothers will be invited to join as they come into proximity. The game canonly be played by users that are in the specific location, which is thecontext defined by the back-end MCRM system. As the user who is in thepark and has had the application downloaded approaches a concessionstand, the ad hoc proximity peer-to-peer system will create a connectionwith the concessions stand's mobile device/and or connected point ofsale (POS) system. The menu from the concessions stand will be displayedon the user's mobile device. The user can select the menu items and theorder will be transmitted to the concessions stand's mobile deviceand/or connected POS system. Once the order has been placed, a cloudbased mobile payment system such as Qualcomm, Inc.'s “PAY” solution ornear field communication (NFC) payment solution such as Google “Wallet”can be used to complete the transaction. When “PAY” is used, the userspurchase is captured and retained in the MCRM system. As a person ismoving around their surroundings inside the park, images can becomeroadmaps and markers. These can offer directional feedback to theend-user. They can be simple logos, arrows, or any images that the IR/ARtechnology recognizes as a directional sign as matched against the cloudbased target database. The image can indicate to one end user to go leftbased on that person's desired destination, the same image can indicateto another user to go right based on the person's desired destinationthat differs from the first person's destination. In certain physicallocations where GPS signals are weak—such as indoors, event—the solutionenables AR/IR targets to become virtual guides and sign posts. Byrecognizing targets that are placed in a known physical location, theMCRM system will capture the target and match it against the targetdatabase. Since the physical location of the target is known, the user'slocation is also known since the user needs to be in front of the targetfor the AR/IR to recognize it. The MCRM system can then send informationabout other known physical places in proximity to the current AR/IRtarget. Information such as directions, rides, distance to these,features of the location/ride and offers from the location can bedisplayed. These will be superimposed on a digital map unlike thetraditional AR experiences where you hold your phone up against thehorizon. These systems are typically dependent on GPS coordinates of theuser and the AR overlays and do not work indoors. The solution uses acombination of the user's known location, the MCRM database of known anddefined locations, the MCRM's knowledge of AR/IR target's locations, andif a known number of these AR/IR targets are recognized as dynamicdirectional markers, directional arrows can appear as the user movesthrough the environment guiding the user to the desired end location. Asthe user moves into an environment where there are display devices thatare connected, a connection directly between the hub and the deviceswill be established by a variety of methods, and information from thehub and the cloud based databases can be displayed on the displays. Thesolution also enables digital signs to display relevant information as auser moves through an environment. The display sign is connected withthe MCRM system in the cloud. Once a user's context bubble enters the“sphere of influence” of the digital sign, the user's mobile device willrecognize the event. The MCRM system will recognize the event from boththe digital display as well as the user. The MCRM will send anappropriate message to be displayed on the display based on the userspast preferences, current state of mind, the user's likely nextdestination as well as any recorded activities in the future. A user'scurrent context can also be defined by the user's location. There arenumerous means to define and record this, and the solution is notdependent upon any specific method. As the user is moving about in thephysical world, the user's context will constantly change. The user canuse the MCRM to specifically record certain aspects of the context andshare it with others. One such embodiment is to place a tag on thespecific location a user is in, and then share this location with otherseither directly to one other person, or to that person through socialmedia, or share with a group of others. Another embodiment of thesolution is the use for live performances such as concerts, circus actsand theater performances inside the park. Aspects within the performancesuch as props, stands, displays, mannequins, statues, dolls can serve asAR markers and pre-recorded acts can be superimposed on them during theperformance. This can be AR overlay that is common and seen the same wayby all in the audience, or it can be customized to be randomly displayedor specifically presented based on the end user's demographics such asage and or gender. The theater or venue that the live performance is incan also be enhanced with AR superimposed content that can be same forall the users in the audience, randomly presented or customized forspecific audience members. Certain locations can be enhanced withcontextually specific features. Offers that are specific to suchlocations can be automatically presented to users that are in such acontext and have a desire to receive them. Within other venues, behaviorthat is beneficial to the venue can be rewarded. Loyalty programs,special front of the line entrance and other VIP treatment can beawarded to users that are in the context and promote to their socialnetwork that they are there. Should their friends respond in a way thatis beneficial to the park owner/operator, such user will receive thebenefits, status and loyalty rewards. The scavenger and treasure huntgames can also be used to control crowds and/or drive traffic to certainparts of the park during certain times. For instance, a treasure huntgame can be initiated at 4 pm and the final clue can be presented rightat the center of the park at 5 pm. As the participants are starting tolook for exits of the park, offers to restaurants they pass on their wayout can be presented as they walk by enticing the visitor to dine at therestaurant rather than exiting and eating outside the park. FIG. [24]“5” shows that as the user is leaving the theme park, rewards forspecific behavior can be presented. The action of leaving can bedetermined by crossing the geo-fence as well as time. Activities such assharing the experience with others, participating in games, can berewarded. The rewards can be for offers to get a product such as parkmerchandise on the way out, or get a bottle of water for the ride home.Information can also be presented at this time such as traffic updates,where the car was parked, crowds on public transportations, altercationsand traffic accidents. FIG. [24] “6” shows when the user is in transiton the way away from the park, information, content and offers can bepresented. They can be a “thank you for coming” note, or offers to go toparticipating restaurants, bars, hotels or retailers on the way home.These can all be sponsored by the theme park or their sponsors. They canbe for products, discount offers or merchandise from the park. FIG. [24]“7” shows that when the user is back at the location as detected in FIG.[24] “1” or another location that has been determined to be the user'shome, additional offers, coupons, information and content can bepresented by the theme park. This can discount offers such as bring afriend and get 50% off, or other enticement offers to return to thepark. This complete solution thereby enables the park owner/operator tocommunicate with their fans from the moment they purchase their tickets,as they are coming to the park, while they are inside the stadium, asthey are returning on the way home and after they have returned home,thus creating loyalty between the fan and the park.

FIG. 25 depicts the solutions described in FIG. 23 and FIG. 24 and showsthe various technologies used in each aspect of the solution. FIG. 25[1] shows that AR and IR is used at the home to capture and verify thatthe user has purchased a ticket. Location and GPS coordinates are usedto verify rate of speed, location and direction in FIG. 25 [2] todetermine that the user is en-route to the location. When arrived,peer-to-peer technologies and near field communication can be used tocreate ad-hoc networks and communication as shown in FIG. 25 [3]. Thesetechnologies can continue to create the experiences while the user is inthe location as shown in FIG. 25 [4] and described in detail in sections[0041] and [0042]. Ad hoc social media and network interaction within-app and in-game rewards can be triggered by location and proximity asthe user is leaving the location as shown in FIG. 25 [5]. As the user isdeparting the location, rate of speed, direction and dynamic locationswill be used to determine that the user is in the process of leaving thelocation as shown in FIG. 25 [6].

FIG. 26 depicts the process used for a scavenger hunt game and treasurehunt game controlled by an MCRM back-end and featuring AR experiences,puzzles, clues and hints. The game starts when a user enters the initialstart zone that has been established as a geo fenced area within theMCRM system. An alert is automatically sent to the mobile device as theuser crosses the fence and the game is triggered and started. The useropens the link and offer that has been pushed to the device by the MCRMsystem. The next AR target is revealed once the user solves the puzzle,hint or quiz that is included in the offer or link. A hint or clue onwhere that next target may be located is presented. The user will thenmove towards that target. An AR compass can be superimposed on theuser's mobile device as the user moves through the real world towardsthe next target. Once there, the user will use the camera on the user'smobile device to seek to locate the next AR target and once found, theimage recognition system on the AR enabled device will recognize thetarget and serve up an AR overlay on the target. This can be another ARpuzzle, quiz, test, hint or offer. Once this is unlocked, the user willbe presented another clue, hint or direction to the next location whereanother AR target and its associated puzzle, quiz or hint is located.The MCRM system maps the user as the user proceeds through the physicalworld, and verifies that the user has been at certain locations byrecording when the AR experiences has been activated as these can onlybe triggered by the target that is located at a specific location. Thesystem will also measure and ensure that the targets are opened in theright sequence. Once the user has competed all the steps, solved all thepuzzles, quizzes and questions, located all the AR targets and activatedall the AR experiences, the user will be presented with a prize and orreward based on their ranking such as time to complete, first tocomplete, effectiveness, shortest route taken or other pre-determinedparameters.

FIG. 27 depicts a user that is wearing a sensor device on the arm thatis able to communicate with the users connected mobile device. Thissensor device is envisioned to measure health and medical stats such asbut not limited to blood pressure, blood sugar levels, cholesterol,heart rate and other medical stats. It is able to communicate with themobile device through Wi-Fi, blue tooth or a combination of othercommunication platforms. There are numerous such devices in themarketplace and more being introduced. In the future, it is envisionedthat these sensors can be embedded in clothing, shoes, Band-Aids,jewelry, accessories and dedicated devices, as well as embedded in theuser's body. The user is about to consume a health related product. Thiscan be medication, vitamins, pills, liquid, healthy juice or otherconsumer products that has a health benefit. Once the user is using ARto trigger an experience that is superimposed on the packaging of thehealth related product, the AR experience triggers the mobile device toactivate the sensor system and record its first set of health data. Thispre-consumption health condition is recorded by the sensor device andtransmitted to the mobile device. Once the user opens the packaging ofthe health related product another AR target is exposed. This can belocated under the lid of a juice bottle or pill bottle. Another ARexperience is superimposed on that target that entertains, informs orencourages the user to consume the product. Shortly after this, the ARexperience will present an activation asking the user to verify thatthey have consumed the product. When this is confirmed and after apreset time, the sensor is reactivated by the mobile device. Anothermeasurement is taken and transmitted to the mobile device, recorded andstored.

FIG. 28 depicts that as the medical sensor device is capturing,recording and transmitting medical and health data to the mobile deviceas described in FIG. 27, the mobile device is also capturing contextualdata at the same time. This contextual data is captured, recorded andstored on the mobile device.

FIG. 29 depicts that the mobile device collects the health data beforeand after the consumption of the health product and combines real timesimultaneously contextual data, records and stores this information. Itthen transmits the health data that has been encrypted to an approvedcloud based database that has been approved to contain, protect andstore medical health records. The contextual data that has been gatheredis transmitted to the MCRM context database. The health data records canbe displayed to authorized medical and health personnel as well as tothe user through other connected devices or the user's mobile devicethrough pre-designed dashboards.

FIG. 30 depicts how the data collected that is stored on the contextualdatabase can be combined and accessed by select individuals andorganizations that are involved in maintaining, monitoring and improvingthe health of the user. These individuals and organizations may includethe user's fitness trainers, their medical doctors as well as theprovider and manufacturer of the product consumed. Each will havevarious levels of access to the contextual and matching medical data.All medical data will be maintain, stored, shared and displayed inaccordance with local and global laws and regulations governing suchissues at the location of the user and the location of the storagefacility. By reviewing the user's context and activity as the user goesthrough the day, behavior and activity levels are captured, recorded andreported to the context CRM system. If the user is not active enough, isnot consuming the health product, is not following health advice, is notgoing to the gym as prescribed, messages can automatically be sent tothe user to entice, remind and encourage the user to change behavior.Conversely, rewards and encouragement can be automatically sent to theusers who participate and is leading a healthy life style. The systemcan also be used to verify that patients are consuming the rightmedication, when they are doing so and in what context they are whiledoing so. Benefits—or lack thereof—from the consumption of the healthproduct can also be measured, verified and recorded. Corrective actionscan be performed if the desired results are not obtained. These can bebehavioral or dosage related.

FIG. 31 depicts an overview of the steps and information flows describedin FIGS. 27, 28, 29 and 30.

FIG. 32 depicts the mobile customer relationship management system(MCRM) customized for an organization that operates a franchise businesswith numerous franchisees at various physical locations. The MCRM masterdashboard interface is created so that only the owner organization hasaccess to the complete solution. This interface controls advertisements,content, offers, merchandise, coupons, games and contests, VIP offersand benefits, rewards, loyalty programs and product offerings such asmenus. The corporation controls the delivery of all of these to theglobal universe of users of the company's enabled apps on the user'smobile devices. The company also can send master and global offers formaster sponsors, such as a franchise organization of fast foodrestaurants can send an offer sponsored by Coca Cola to all its mobileapp users. The organization will also enable its franchisees to access asub directory of the MCRM with limited features and abilities. Thesewill have restrictions such as only be able to deliver messages in aspecific geographical area. They can deliver experiences that aretargeted and only enabled inside the franchisees own restaurants. Theycan deliver messages and offers for local sponsors. The franchisororganization has real time overview of the activities and offers thatthe franchisee sends, with immediate reporting and oversight. Anyunwanted, ineffective or incorrect activity can immediately beterminated via a “kill switch” feature in the system that willimmediately suspend the activity and replace it with approved offers andcontent. The master franchise owner can also send messages and recordwhen a user is in any of their restaurants and can communicate withthose at that very moment. The users in the restaurant will therefore beable to be rewarded with an enhanced dining experience that is locallytailored as well as managed, approved and supervised by the franchiseorganization.

FIG. 33 depicts the mobile customer relationship management system(MCRM) customized for a sport league that operates a franchise likebusiness with numerous sport team franchisees at various physicallocations. The MCRM master dashboard interface is created so that onlythe owner organization has access to the complete solution. Thisinterface controls advertisements, content, offers, merchandise,coupons, games and contests, VIP offers and benefits, rewards, loyaltyprograms and product offerings such as league wide game schedules andfantasy leagues. The corporation controls the delivery of all of theseto the global universe of the sport fans using the leagues enabled appson the user's mobile devices. The league also can send master and globaloffers for master sponsors, such as a franchise organization such as theNFL can send an offer sponsored by Papa Johns to all its mobile appusers. The organization will also enable its franchisees to access a subdirectory of the MCRM with limited features and abilities. These willhave restrictions such as only be able to deliver messages in a specificgeographical area and to confirmed and verified fans in other cities.They can deliver experiences that are targeted and only enabled insidethe team's stadiums. They can deliver messages and offers for localsponsors. The league has real time overview of the activities and offersthat the teams send, with immediate reporting and oversight. Anyunwanted, ineffective or incorrect activity can immediately beterminated via a “kill switch” feature in the system that willimmediately suspend the activity and replace it with approved offers andcontent. The league can also send messages and record when a user is inany of the stadiums and can communicate with those at that very moment.The fans in the stands will therefore be able to be rewarded with anenhanced stadium experience that is locally tailored as well as managed,approved and supervised by the league. Compete stadium enhancedexperiences depicted in FIG. 23 and described in section [0041] can bedelivered, controlled and monitored using this system.

As may be appreciated from the above written description and figures thenumerous systems, architectures and processes have been described andare summarized as follows:

1. Overall Architecture of Contextually intelligent communicationplatform.

2. Personal communication hub.

3. Data collected by devices and sensors in an area surrounding theuser, captured by and on a communication device.

4. Collecting contextual data and replaying the data later.

5. Using distance between an AR target and the device overlaying the ARexperience as a means to control the volume of sound and the level ofother feedback system.

6. Using AR and IR as a means to detect, verify and capture a user'scontext.

7. Using AR and IR as a means to detect, verify and record theeffectiveness of advertising.

8. Using AR and IR for location based scavenger and treasure hunts.

9. Using AR and IR for location based scavenger and treasure hunts withpuzzles and quizzes.

10. Using AR and IR for location based scavenger and treasure hunts withpuzzles and quizzes that unlocks coupons and offers when completed.

11. Sharing a user's context using social networks.

12. AR targets that are discovered and made available to select user.

13. Sensing and monitoring context for law enforcement and militarywhere detectable events such as gunshot triggers an alarm whilecapturing the user's real-time context.

14. Using AR and IR as a means to detect, verify and record what theuser is watching on TV.

15. Using AR and IR as a means to detect, verify and record what theuser is watching on TV, record and verify the user's context at thatmoment.

16. Using AR and IR as a means to detect, verify and record what theuser is watching on TV, record and verify the user's context at thatmoment, then present additional information that is relevant to what ison the TV directly to the user's mobile device.

17. Using AR and IR as a means to detect, verify and record what theuser is watching on TV, record and verify the user's context at thatmoment, then offer alternative and/or relevant content and present thisdirectly to the user's mobile device.

18. Using aspects of merchandise such as product labels and hang tags asvirtual displays via AR overlays enabling the designer, manufacturer orowner of the product to present, demonstrate, showcase, explain andmarket the product directly to an end user's mobile device.

19. Means of using location as a trigger for communication between amobile device and a retailer's POS systems via cloud or peer to peercommunication.

20. Means of using location as a trigger for communication between amobile device and a retailer's POS systems via cloud or peer to peercommunication to initiate a mobile payment transaction.

21. Using proximity based and peer-to-peer communication to initiate andenable location specific game play between user's mobile devices whilein a specific location.

22. Using proximity based and peer-to-peer communication to initiate andenable location specific game play between user's mobile devices whilein a specific location and rewarding the winner with a location specificprize or benefit.

23. The use of images and targets that triggers dynamic, interactive andpersonalized directional signs and guidance in AR enabling navigation.

24. Development of specific fonts, images and signs that are efficientlyrecognized by IR and can be interpreted to create dynamic andpersonalized AR experiences.

25. Method to customize and personalize static print, products, imagesusing AR and MCRM.

26. Sensor network worn by participants of sporting events used tore-create the game experience to viewers of the game.

27. G-force and impact sensor, strap connector sensor and communicationdevice on sporting helmets to measure impact and warn if strap is not onor when impacts have exceeded acceptable levels.

28. Method for re-creating real life sport events in AR representationof the game.

29. AR use for corporate promotions and message delivery.

30. 3-D AR representation of complex organizational charts.

31. Capturing context to enable communication between devices in theproximity to each other.

32. Method of communication between a sport franchise team and its fansbefore, during and after a sporting event to enhance the fan experienceand secure fan loyalty.

33. Method of communication between a retailer and its customers before,during and after the customer comes to the retailer's location toenhance the shopping experience and establish loyalty and frequentpurchases.

34. Game play on mobile devices in specific locations such as stadiumswith results replayed and displayed on large in stadium monitors.

35. Using location and proximity to deliver exclusive content to amobile device that is in a specific location.

36. Using AR to create virtual photos and rewarding the users who sharethese on social networks.

37. Using peer-to-peer communication to present a restaurant'sinteractive menus to mobile devices that are in proximity to therestaurant.

38. Using peer-to-peer communication to present a restaurant'sinteractive menus to mobile devices that are in proximity to therestaurant, enabling the mobile device to place the order and pay forthe order, communication with the restaurant's order taking system viapeer to peer, mobile network, near field communication or Wi-Fi.

39. Using AR to present a story that is initiated off an advertisement,continue the story triggered by AR targets inside a retailer andcompleting the story off AR triggers inside the product.

40. Using AR to tag items and merchandise on a user's virtual wish list.

41. Using AR to identify items and merchandise that is on a differentuser's virtual wish list.

42. Using AR and mobile devices ability to detect location to verifythat an advertisement has succeeded in presenting a viewer of theadvertisement to the location that was in the advertisement.

43. Using AR and scavenger hunt games for crowd control and management.

44. UI: AR half mode—superimposing a virtual game on top of the realworld without the use of markers.

45. UI: AR search mode—display a transparent representation of thetarget, once a match has been found, create “lock-on” animation.

46. UI: AR porting mode—once a target has been found, the AR experienceis initiated, and then when the target is lost, the AR game is portedupon the device and continues until the target is re-connected.

47. AR Posters; printed posters that are personalized for users via ARexperiences.

48. AR Posters; printed posters that feature call to action, buy now andinformation such as locations of theaters nearby showing a movie, thetimes of the showing, movie trailer and the ability to purchase tickets.

49. AR trading cards; avatars are superimposed in AR with characteristicand features based on dynamic data such as NFL player's real life stats.

50. AR targets; using a frame to create an AR experience that issuperimposed upon a user's drawn representation of the AR.

51. AR education; using AR 3-d puzzles to educate and teach.

52. AR education; using AR 3-d puzzles to educate and teach, recordprogress, report results to teacher and caregiver.

53. AR social games; using one target as the trigger for a game playedby users in proximity to each other.

54. AR books; enhancing books with AR such as 3D games, puzzles, quizzesand tests, recording the interactions, rewarding for success.

55. AR weather forecast; using static weather page in newspapers topresent AR overlay that is hyper local for the user's current location.

56. AR dynamic print; using AR to update and present dynamicallypersonalize content on printed materials.

57. AR dynamic catalogs; using AR generated avatars as guides andpersonalized. shopping assistants in catalogs, with buyingrecommendations, wish lists and retailer interaction.

58. AR communication channel; the use of printed material such as ads,markers on clothing, product packaging to present dynamic andpersonalized content, offers, entertainment and information.

59. AR maps; use of traditional maps with AR enhancements to showcaselocations.

60. AR interactive maps; use of traditional maps with interactivedynamic and personalized AR content including directional overlays, “youare here” pin points and highlights along the selected route with themap ported on the device when tracking is lost.

61. Contextual check-in: once a user enters a context where there is acheck-in required, this is performed automatically.

62. AR for indoor navigation; use of AR targets enables dynamicinteractive and personalized navigation.

63. Personal communication hub; central hub that collect informationfrom various sensor systems and displays information on a selection ofconnected display devices.

64. Personal communication hub; central hub that collect informationfrom various sensor systems and stores information on a cloud basedserver.

65. Digital signs; display signs presents digital personalized contentthat is appropriate for users that are close by.

66. Contextual search; the method to search for physical location thathas the desired context.

67. Health; Use of AR, IR and contextual awareness to measure, track andverify the consumption of health and medical related products.

68. Health; Use of AR, IR and contextual awareness and sensors worn bythe user to measure, track and verify the consumption of health andmedical related products and their effect on the user.

69. Health; Use of AR, IR and contextual awareness to measure, track andverify the consumption of health and medical related products and verifythe behavior of the user while offering advice, recommendations andreminders about behaviors that are beneficial to the user.

70. Health; Use of AR, IR and contextual awareness to measure, track andverify the consumption of health and medical related products and verifythe behavior of the user while offering advice, recommendations andreminders about behaviors that are beneficial to the user storing allhealth related information on compliant servers.

71. Context replay; the ability to record a user's actions, activitiesand context during a time period for replay and review at a later time.

72. Contextual AR; the ability to superimpose context upon digitalavatars that replay the context in AR.

73. Hyper Location; the use of known sensor systems in a knownenvironment to define, verify and report a specific location.

74. Hyper Location; the use of known sensor systems in a knownenvironment such as a motor vehicle to define, verify and report aspecific location inside the vehicle such as driver seat

75. Sharing of Contextual information; the ability to share the contexta user is currently in or was in at a given moment of time when thecontext was recorded.

76. AR overlay of environments; interiors and locations that areenhanced by AR that can be same for all, or customized and personalizedto adapt to specific user as they enter the environment.

77. AR enhancement of events and performances; use of AR targets in liveand recorded performances.

78. Location based and contextually restrictive features on mobilephones; betting and wagers at specific locations by authorized users.

79. Location based and contextually dependent rewards; based onlocation, context and the user's sharing of this information,presentation of offers, rewards and loyalty benefits and privileges thatare relevant to the location and context based on the user's promotionof the experience through social networks.

80. Context recording; the future use of context recording and replay inspecific environments.

81. Context alerts; method for defining context where the user of amobile device notify that the user does not want to be interrupted.

82. Contextually enhanced 3D video recording and replay; by capturingthe context of a user created 3D video, the replay of said 3D video in acontextual replay environment will include the context that was capturedwhile recording.

83. Use of a contextually intelligent mobile marketing system for fundraising efforts.

84. Contextual Intelligence; method for developing.

85. Contextual Wisdom; method for developing.

86. Master system for a franchisor to manage communication with its endcustomers while enabling individual franchisees to communicate withcustomers in their geographical area and locations under the oversightand control of the master franchisor.

87. Master system for a sports league to manage communication with itsfans of the sports league while enabling individual teams to communicatewith their fans in their geographical area and locations as well as teamfans in other areas under the oversight and control of the master sportleague owner.

What is claimed is:
 1. A computer-implemented process for contextuallyintelligent mobile communication comprising: providing a mobilecommunication device; providing a plurality of sensors residing in saiddevice; associating said device with a first user; providing first userpast and historical data; generating contextual data for said first userwith said device; gathering said contextual data on said device tocreate first user current contextual data; uploading said first usercurrent contextual data via a wide area mobile communication network toa contextually intelligent server; matching said first user currentcontextual data with said first user past and historical data;generating feedback data for said first user that is contextuallyrelevant to said user's current context and predictably useful to saiduser as said user enters a new context to form first user usefulfeedback data; transmitting to and displaying on said device said firstuser useful feedback data to provide said contextually intelligentmobile communication.